[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:34 dist-test-slave-qfk4 krb5kdc[7762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:34 dist-test-slave-qfk4 krb5kdc[7762](info): set up 2 sockets
Jun 19 06:39:34 dist-test-slave-qfk4 krb5kdc[7762](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 06:39:35.469703 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.005s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:39:35 dist-test-slave-qfk4 krb5kdc[7762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:39:35.626001 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40905
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:40905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:35.934134 7775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:35.934433 7775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:35.934551 7775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:35.941962 7775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:35.942078 7775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:35.942135 7775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:35.942216 7775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:35.942281 7775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:35.952310 7775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:40905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:35.953689 7775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:35.955118 7775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:35.967084 7780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:35.967075 7781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:35.967264 7783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:35.967958 7775 server_base.cc:1034] running on GCE node
I20260619 06:39:35.968777 7775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:35.970304 7775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:35.971529 7775 hybrid_clock.cc:648] HybridClock initialized: now 1781851175971468 us; error 55 us; skew 500 ppm
I20260619 06:39:35.971895 7775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:35 dist-test-slave-qfk4 krb5kdc[7762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851175, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:35.975042 7775 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:39:35.976663 7775 webserver.cc:460] Webserver started at http://127.7.146.254:33393/ using document root <none> and password file <none>
I20260619 06:39:35.977362 7775 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:35.977478 7775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:35.977759 7775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:35.981124 7775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "70485fceb39d4cde98ed52b7f5f544db"
format_stamp: "Formatted at 2026-06-19 06:39:35 on dist-test-slave-qfk4"
server_key: "e71e31b4c53804f8a579b381cf1b3faa"
server_key_iv: "68a40ed421e235f8299b21f570c54038"
server_key_version: "encryptionkey@0"
I20260619 06:39:35.981864 7775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "70485fceb39d4cde98ed52b7f5f544db"
format_stamp: "Formatted at 2026-06-19 06:39:35 on dist-test-slave-qfk4"
server_key: "e71e31b4c53804f8a579b381cf1b3faa"
server_key_iv: "68a40ed421e235f8299b21f570c54038"
server_key_version: "encryptionkey@0"
I20260619 06:39:35.987085 7775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:39:35.990567 7790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:35.991895 7775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 06:39:35.992054 7775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "70485fceb39d4cde98ed52b7f5f544db"
format_stamp: "Formatted at 2026-06-19 06:39:35 on dist-test-slave-qfk4"
server_key: "e71e31b4c53804f8a579b381cf1b3faa"
server_key_iv: "68a40ed421e235f8299b21f570c54038"
server_key_version: "encryptionkey@0"
I20260619 06:39:35.992228 7775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:36.020722 7775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:36.025177 7775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:36.025454 7775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:36.042893 7775 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:40905
I20260619 06:39:36.042901 7841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:40905 every 8 connection(s)
I20260619 06:39:36.044222 7775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:36.048244 7842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:36.051383 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 7775
I20260619 06:39:36.051745 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:39:36.054206 7755 external_mini_cluster.cc:1428] Setting key cd341b9eef122ed28f5399abe5311580
I20260619 06:39:36.060907 7842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db: Bootstrap starting.
I20260619 06:39:36.063807 7842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:36.064803 7842 log.cc:824] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:36.067380 7842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db: No bootstrap required, opened a new log
I20260619 06:39:36.076973 7842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70485fceb39d4cde98ed52b7f5f544db" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40905 } }
I20260619 06:39:36.077325 7842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:36.077425 7842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70485fceb39d4cde98ed52b7f5f544db, State: Initialized, Role: FOLLOWER
I20260619 06:39:36.078016 7842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70485fceb39d4cde98ed52b7f5f544db" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40905 } }
I20260619 06:39:36.078231 7842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:36.078357 7842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:36.078522 7842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:36.080828 7842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70485fceb39d4cde98ed52b7f5f544db" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40905 } }
I20260619 06:39:36.081296 7842 leader_election.cc:302] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70485fceb39d4cde98ed52b7f5f544db; no voters:
I20260619 06:39:36.081881 7842 leader_election.cc:288] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:36.081987 7847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [term 1 FOLLOWER]: Leader election won for term 1
Jun 19 06:39:36 dist-test-slave-qfk4 krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:36.082968 7847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [term 1 LEADER]: Becoming Leader. State: Replica: 70485fceb39d4cde98ed52b7f5f544db, State: Running, Role: LEADER
I20260619 06:39:36.083657 7847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70485fceb39d4cde98ed52b7f5f544db" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40905 } }
I20260619 06:39:36.084113 7842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:36.086647 7848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70485fceb39d4cde98ed52b7f5f544db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70485fceb39d4cde98ed52b7f5f544db" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40905 } } }
I20260619 06:39:36.087016 7848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:36.088482 7849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70485fceb39d4cde98ed52b7f5f544db. Latest consensus state: current_term: 1 leader_uuid: "70485fceb39d4cde98ed52b7f5f544db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70485fceb39d4cde98ed52b7f5f544db" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40905 } } }
I20260619 06:39:36.088838 7849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:36.092648 7852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:36.095309 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:36.058834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.7.146.254:40905)
0619 06:39:36.059702 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:36.059717 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:36.061087 (+ 1370us) server_negotiation.cc:407] Connection header received
0619 06:39:36.069708 (+ 8621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:36.069735 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:36.070233 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:36.070681 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:36.073485 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:36.075581 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:36.077553 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:36.078261 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:36.078457 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:36.084417 (+ 5960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:36.084426 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:36.084442 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:36.084485 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:36.088625 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:36.089921 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:36.089931 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:36.089941 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:36.090175 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:36.090789 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:36.090795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:36.090799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:36.091634 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:36.092855 (+ 1221us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:36.093198 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 06:39:36.093704 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":389,"threads_started":1}
I20260619 06:39:36.098155 7852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:36.109495 7852 catalog_manager.cc:1347] Generated new cluster ID: c637bc45dfcc4bcfb2fc89dd1212b4b9
I20260619 06:39:36.109617 7852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:36.152380 7852 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:36.154273 7852 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:36.165074 7852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70485fceb39d4cde98ed52b7f5f544db: Generated new TSK 0
I20260619 06:39:36.166111 7852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:39:36.212177 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40905
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:36.527041 7868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:36.527347 7868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:36.527467 7868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:36.534734 7868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:36.534853 7868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:36.534998 7868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:39:36.544796 7868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:36.546063 7868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:36.547441 7868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:36.559998 7874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:36.560042 7873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:36.559986 7876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:36.560814 7868 server_base.cc:1034] running on GCE node
I20260619 06:39:36.561345 7868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:36.562263 7868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:36.563467 7868 hybrid_clock.cc:648] HybridClock initialized: now 1781851176563393 us; error 66 us; skew 500 ppm
I20260619 06:39:36.563808 7868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:36 dist-test-slave-qfk4 krb5kdc[7762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851176, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:36.566815 7868 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:39:36.568153 7868 webserver.cc:460] Webserver started at http://127.7.146.193:40755/ using document root <none> and password file <none>
I20260619 06:39:36.568842 7868 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:36.568940 7868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:36.569186 7868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:36.571925 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "5e5e41a786d24a1eaab9163881ffb1bf"
format_stamp: "Formatted at 2026-06-19 06:39:36 on dist-test-slave-qfk4"
server_key: "4e3c2269f3dbc576c42fedc57c95d6cb"
server_key_iv: "1d0de3c9faa567a5306e30cf53034fd7"
server_key_version: "encryptionkey@0"
I20260619 06:39:36.572670 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "5e5e41a786d24a1eaab9163881ffb1bf"
format_stamp: "Formatted at 2026-06-19 06:39:36 on dist-test-slave-qfk4"
server_key: "4e3c2269f3dbc576c42fedc57c95d6cb"
server_key_iv: "1d0de3c9faa567a5306e30cf53034fd7"
server_key_version: "encryptionkey@0"
I20260619 06:39:36.577696 7868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:39:36.581048 7883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:36.582430 7868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:39:36.582577 7868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "5e5e41a786d24a1eaab9163881ffb1bf"
format_stamp: "Formatted at 2026-06-19 06:39:36 on dist-test-slave-qfk4"
server_key: "4e3c2269f3dbc576c42fedc57c95d6cb"
server_key_iv: "1d0de3c9faa567a5306e30cf53034fd7"
server_key_version: "encryptionkey@0"
I20260619 06:39:36.582752 7868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:36.606906 7868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:36.611111 7868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:36.611389 7868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:36.612344 7868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:39:36.613708 7868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:36.613778 7868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:36.613888 7868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:36.613940 7868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:36.643424 7868 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:34299
I20260619 06:39:36.643494 7995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:34299 every 8 connection(s)
I20260619 06:39:36.644707 7868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:39:36.647429 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 7868
I20260619 06:39:36.647637 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:39:36.648382 7755 external_mini_cluster.cc:1428] Setting key 64160843d9f1ef5cee05c7ef56bffce1
Jun 19 06:39:36 dist-test-slave-qfk4 krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851176, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
I20260619 06:39:36.664333 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:36.647227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:36259 (local address 127.7.146.254:40905)
0619 06:39:36.647700 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:36.647705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:36.648885 (+ 1180us) server_negotiation.cc:407] Connection header received
0619 06:39:36.650930 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:36.650935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:36.651014 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:36.651102 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:36.652594 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:36.653321 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:36.654792 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:36.655181 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:36.655286 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:36.658605 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:36.658610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:36.658614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:36.658644 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:36.661214 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:36.662024 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:36.662029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:36.662032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:36.662156 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:36.662635 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:36.662639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:36.662641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:36.662928 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:36.663041 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:36.663791 (+ 750us) server_negotiation.cc:299] Negotiation successful
0619 06:39:36.663985 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
I20260619 06:39:36.665396 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:36.647869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40905 (local address 127.7.146.193:36259)
0619 06:39:36.648705 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:36.648750 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:36.650671 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:36.651346 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:36.651355 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:36.651840 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:36.652406 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:36.652424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:36.653461 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:36.653465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:36.654657 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:36.654669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:36.655383 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:36.655391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:36.655605 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:36.656223 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:36.656245 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:36.658386 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:36.661356 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:36.661365 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:36.661384 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:36.661886 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:36.662323 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:36.662326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:36.662329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:36.662512 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:36.663064 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:36.663070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:36.663410 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 06:39:36.663644 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 06:39:36.664120 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":324,"threads_started":1}
I20260619 06:39:36.666896 7996 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40905
I20260619 06:39:36.667220 7996 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:36.668009 7996 heartbeater.cc:511] Master 127.7.146.254:40905 requested a full tablet report, sending...
I20260619 06:39:36.669994 7807 ts_manager.cc:194] Registered new tserver with Master: 5e5e41a786d24a1eaab9163881ffb1bf (127.7.146.193:34299)
I20260619 06:39:36.671406 7807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:36259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:39:36.684487 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40905
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:36.993517 8002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:36.993811 8002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:36.993944 8002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:37.001245 8002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:37.001366 8002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:37.001518 8002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:39:37.011509 8002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:37.013018 8002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:37.014513 8002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:37.027760 8010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:37.027822 8007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:37.027786 8008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:37.028637 8002 server_base.cc:1034] running on GCE node
I20260619 06:39:37.029218 8002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:37.030234 8002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:37.031481 8002 hybrid_clock.cc:648] HybridClock initialized: now 1781851177031432 us; error 37 us; skew 500 ppm
I20260619 06:39:37.031845 8002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:37 dist-test-slave-qfk4 krb5kdc[7762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851177, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:37.034869 8002 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:39:37.036309 8002 webserver.cc:460] Webserver started at http://127.7.146.194:44387/ using document root <none> and password file <none>
I20260619 06:39:37.036998 8002 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:37.037112 8002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:37.037401 8002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:37.040167 8002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "9d8789e102014e00b8fad14814d48be4"
format_stamp: "Formatted at 2026-06-19 06:39:37 on dist-test-slave-qfk4"
server_key: "766c71563f458bcb4c9d2d399a8819bf"
server_key_iv: "aac9d9a5fbc62519bba05f08f6480ba6"
server_key_version: "encryptionkey@0"
I20260619 06:39:37.040941 8002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "9d8789e102014e00b8fad14814d48be4"
format_stamp: "Formatted at 2026-06-19 06:39:37 on dist-test-slave-qfk4"
server_key: "766c71563f458bcb4c9d2d399a8819bf"
server_key_iv: "aac9d9a5fbc62519bba05f08f6480ba6"
server_key_version: "encryptionkey@0"
I20260619 06:39:37.046128 8002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 06:39:37.049613 8017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:37.051013 8002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:39:37.051160 8002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "9d8789e102014e00b8fad14814d48be4"
format_stamp: "Formatted at 2026-06-19 06:39:37 on dist-test-slave-qfk4"
server_key: "766c71563f458bcb4c9d2d399a8819bf"
server_key_iv: "aac9d9a5fbc62519bba05f08f6480ba6"
server_key_version: "encryptionkey@0"
I20260619 06:39:37.051348 8002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:37.070211 8002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:37.074347 8002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:37.074612 8002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:37.075534 8002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:39:37.076856 8002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:37.076929 8002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:37.077045 8002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:37.077095 8002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:37.106513 8002 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:38727
I20260619 06:39:37.106556 8129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:38727 every 8 connection(s)
I20260619 06:39:37.107725 8002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:39:37.109591 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 8002
I20260619 06:39:37.109782 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:39:37.110211 7755 external_mini_cluster.cc:1428] Setting key 5c465b7c156fa1e166b70713b0a23395
Jun 19 06:39:37 dist-test-slave-qfk4 krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851177, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
I20260619 06:39:37.129899 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:37.110370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:33107 (local address 127.7.146.254:40905)
0619 06:39:37.110568 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:37.110574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:37.112183 (+ 1609us) server_negotiation.cc:407] Connection header received
0619 06:39:37.114257 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:37.114263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:37.114356 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:37.114465 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:37.116077 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.117149 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.118768 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.119395 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.119515 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:37.123675 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:37.123682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:37.123686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:37.123724 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:37.126421 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.127333 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.127340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.127343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.127514 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.127970 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.127976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.127979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.128361 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:37.128512 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:37.129323 (+ 811us) server_negotiation.cc:299] Negotiation successful
0619 06:39:37.129546 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:39:37.130779 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:37.111079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40905 (local address 127.7.146.194:33107)
0619 06:39:37.112001 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:37.112061 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:37.113941 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:37.114765 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:37.114776 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:37.115315 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:37.115906 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:37.115923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.117311 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.117315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:37.118629 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:37.118641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.119604 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.119609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:37.119866 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:37.120621 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:37.120653 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:37.123385 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:37.126573 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:37.126581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:37.126595 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:37.127185 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:37.127657 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:37.127661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:37.127664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:37.127857 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:37.128498 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:37.128505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:37.128963 (+ 458us) client_negotiation.cc:769] Sending connection context
0619 06:39:37.129219 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 06:39:37.129620 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":362,"threads_started":1}
I20260619 06:39:37.132333 8130 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40905
I20260619 06:39:37.132668 8130 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:37.133437 8130 heartbeater.cc:511] Master 127.7.146.254:40905 requested a full tablet report, sending...
I20260619 06:39:37.134941 7807 ts_manager.cc:194] Registered new tserver with Master: 9d8789e102014e00b8fad14814d48be4 (127.7.146.194:38727)
I20260619 06:39:37.136047 7807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:33107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:39:37.146261 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40905
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:37.457202 8135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:37.457479 8135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:37.457579 8135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:37.465121 8135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:37.465255 8135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:37.465376 8135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:39:37.475255 8135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40905
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:37.476466 8135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:37.477758 8135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:37.490612 8141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:37.490643 8140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:37.490581 8143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:37.491575 8135 server_base.cc:1034] running on GCE node
I20260619 06:39:37.492144 8135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:37.493126 8135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:37.494352 8135 hybrid_clock.cc:648] HybridClock initialized: now 1781851177494291 us; error 43 us; skew 500 ppm
I20260619 06:39:37.494715 8135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:37 dist-test-slave-qfk4 krb5kdc[7762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851177, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:37.498008 8135 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:39:37.499526 8135 webserver.cc:460] Webserver started at http://127.7.146.195:46737/ using document root <none> and password file <none>
I20260619 06:39:37.500250 8135 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:37.500360 8135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:37.500705 8135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:37.503746 8135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "3a57c2e2d47d46578b0e1fc5e8928454"
format_stamp: "Formatted at 2026-06-19 06:39:37 on dist-test-slave-qfk4"
server_key: "c8b1c0eefe640568bcaf715e97f22fb0"
server_key_iv: "64ee4e82d33e9366f66b83dad34af8ce"
server_key_version: "encryptionkey@0"
I20260619 06:39:37.504487 8135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "3a57c2e2d47d46578b0e1fc5e8928454"
format_stamp: "Formatted at 2026-06-19 06:39:37 on dist-test-slave-qfk4"
server_key: "c8b1c0eefe640568bcaf715e97f22fb0"
server_key_iv: "64ee4e82d33e9366f66b83dad34af8ce"
server_key_version: "encryptionkey@0"
I20260619 06:39:37.509748 8135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:39:37.513327 8150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:37.514799 8135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:37.514938 8135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "3a57c2e2d47d46578b0e1fc5e8928454"
format_stamp: "Formatted at 2026-06-19 06:39:37 on dist-test-slave-qfk4"
server_key: "c8b1c0eefe640568bcaf715e97f22fb0"
server_key_iv: "64ee4e82d33e9366f66b83dad34af8ce"
server_key_version: "encryptionkey@0"
I20260619 06:39:37.515148 8135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:37.539301 8135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:37.543258 8135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:37.543515 8135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:37.544457 8135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:39:37.545750 8135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:37.545828 8135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:37.545941 8135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:37.545989 8135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:37.574390 8135 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:39013
I20260619 06:39:37.574446 8262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:39013 every 8 connection(s)
I20260619 06:39:37.575652 8135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:39:37.581576 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 8135
I20260619 06:39:37.581779 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:39:37.582191 7755 external_mini_cluster.cc:1428] Setting key e29beac4d44e2f4296855b74bdd8059a
Jun 19 06:39:37 dist-test-slave-qfk4 krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851177, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:37.594806 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:37.578266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:42043 (local address 127.7.146.254:40905)
0619 06:39:37.578446 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:37.578450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:37.579610 (+ 1160us) server_negotiation.cc:407] Connection header received
0619 06:39:37.581459 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:37.581464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:37.581552 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:37.581667 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:37.583172 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.584159 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.586163 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.586585 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.586674 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:37.589901 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:37.589906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:37.589910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:37.589946 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:37.592207 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.592923 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.592927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.592929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.593035 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.593362 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.593365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.593367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.593614 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:37.593725 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:37.594381 (+ 656us) server_negotiation.cc:299] Negotiation successful
0619 06:39:37.594540 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:39:37.595835 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:37.578768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40905 (local address 127.7.146.195:42043)
0619 06:39:37.579446 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:37.579490 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:37.581178 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:37.581929 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:37.581937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:37.582463 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:37.583021 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:37.583038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.584342 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.584346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:37.586038 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:37.586050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.586755 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.586760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:37.586952 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:37.587551 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:37.587572 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:37.589684 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:37.592338 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:37.592344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:37.592364 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:37.592805 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:37.593112 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:37.593115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:37.593117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:37.593284 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:37.593710 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:37.593715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:37.594085 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 06:39:37.594302 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 06:39:37.594688 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":258,"threads_started":1}
I20260619 06:39:37.597451 8263 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40905
I20260619 06:39:37.597730 8263 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:37.598419 8263 heartbeater.cc:511] Master 127.7.146.254:40905 requested a full tablet report, sending...
I20260619 06:39:37.599660 7806 ts_manager.cc:194] Registered new tserver with Master: 3a57c2e2d47d46578b0e1fc5e8928454 (127.7.146.195:39013)
I20260619 06:39:37.600505 7806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:42043
I20260619 06:39:37.610616 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:39:37 dist-test-slave-qfk4 krb5kdc[7762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:39:37 dist-test-slave-qfk4 krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:39:37.651595 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:37.636177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.7.146.193:34299)
0619 06:39:37.636831 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:37.636839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:37.636873 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:39:37.637094 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:37.637101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:37.637331 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:37.637484 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:37.638843 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.639836 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.641150 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.641716 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.641829 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:37.645858 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:37.645869 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:37.645881 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:37.645923 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:37.648960 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.649605 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.649612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.649614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.649737 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.650094 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.650097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.650099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.650421 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:37.650692 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:37.650994 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 06:39:37.651250 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":252,"threads_started":1}
W20260619 06:39:37.652868 7911 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60454
Jun 19 06:39:37 dist-test-slave-qfk4 krb5kdc[7762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260619 06:39:37.674863 7996 heartbeater.cc:503] Master 127.7.146.254:40905 was elected leader, sending a full tablet report...
Jun 19 06:39:37 dist-test-slave-qfk4 krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:39:37.693650 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:37.679736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.7.146.193:34299)
0619 06:39:37.679985 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:37.679991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:37.680308 (+ 317us) server_negotiation.cc:407] Connection header received
0619 06:39:37.680444 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:37.680447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:37.680541 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:37.680667 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:37.682100 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.682820 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.684171 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.684629 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.684752 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:37.688832 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:37.688862 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:37.688866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:37.688903 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:37.691210 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.691821 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.691827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.691829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.691952 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.692290 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.692295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.692297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.692585 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:37.692811 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:37.693132 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 06:39:37.693345 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 06:39:37.695012 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 7868
I20260619 06:39:37.707535 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:37.696054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.7.146.193:34299)
0619 06:39:37.696238 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:37.696243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:37.696435 (+ 192us) server_negotiation.cc:407] Connection header received
0619 06:39:37.696626 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:37.696632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:37.696695 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:37.696772 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:37.697918 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.698696 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.699903 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.700317 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.700463 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:37.702626 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:37.702632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:37.702637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:37.702668 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:37.705141 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.705732 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.705738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.705742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.705869 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.706194 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.706200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.706203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.706495 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:37.706695 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:37.707003 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 06:39:37.707198 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:39:37.708256 7991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:37.790791 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 7868
I20260619 06:39:37.802877 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 8002
Jun 19 06:39:37 dist-test-slave-qfk4 krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:39:37.818689 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:37.803655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.7.146.194:38727)
0619 06:39:37.804278 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:37.804285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:37.804311 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:39:37.804377 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:37.804384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:37.804643 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:37.804786 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:37.805957 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.807077 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.808346 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.809105 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.809222 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:37.813028 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:37.813038 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:37.813052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:37.813089 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:37.815835 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.816508 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.816515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.816517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.816816 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.817208 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.817212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.817215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.817574 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:37.817855 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:37.818191 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 06:39:37.818404 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":341,"threads_started":1}
I20260619 06:39:37.820087 8125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:37.901131 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 8002
I20260619 06:39:37.913002 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 8135
Jun 19 06:39:37 dist-test-slave-qfk4 krb5kdc[7762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:39:37.928804 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:37.914189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.7.146.195:39013)
0619 06:39:37.914810 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:37.914817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:37.914849 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:39:37.914913 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:37.914920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:37.915157 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:37.915322 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:37.916513 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.917556 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.918778 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:37.919380 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:37.919473 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:37.923457 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:37.923468 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:37.923483 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:37.923522 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:37.926263 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.926927 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.926933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.926936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.927051 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:37.927433 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:37.927436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:37.927438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:37.927753 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:37.927976 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:37.928241 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 06:39:37.928436 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":295,"threads_started":1}
I20260619 06:39:37.930090 8258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:38.012398 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 8135
I20260619 06:39:38.025187 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 7775
I20260619 06:39:38.026418 7837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:38.127064 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 7775
2026-06-19T06:39:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3669 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:38 dist-test-slave-qfk4 krb5kdc[8284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:38 dist-test-slave-qfk4 krb5kdc[8284](info): set up 2 sockets
Jun 19 06:39:38 dist-test-slave-qfk4 krb5kdc[8284](info): commencing operation
krb5kdc: starting...
W20260619 06:39:39.128260 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:39:39 dist-test-slave-qfk4 krb5kdc[8284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:39:39.287927 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:44155
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:44155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:39.596931 8297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:39.597237 8297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:39.597370 8297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:39.604681 8297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:39.604820 8297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:39.604899 8297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:39.604979 8297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:39.605041 8297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:39.615064 8297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:44155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:44155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:39.616420 8297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:39.617733 8297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:39.629038 8302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:39.629005 8305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:39.629077 8303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:39.629755 8297 server_base.cc:1034] running on GCE node
I20260619 06:39:39.630530 8297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:39.631938 8297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:39.633138 8297 hybrid_clock.cc:648] HybridClock initialized: now 1781851179633080 us; error 51 us; skew 500 ppm
I20260619 06:39:39.633478 8297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:39 dist-test-slave-qfk4 krb5kdc[8284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851179, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:39.641261 8297 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:39:39.642913 8297 webserver.cc:460] Webserver started at http://127.7.146.254:34697/ using document root <none> and password file <none>
I20260619 06:39:39.643618 8297 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:39.643733 8297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:39.644021 8297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:39.647017 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "8304899539bf4d5e87578ff990a95864"
format_stamp: "Formatted at 2026-06-19 06:39:39 on dist-test-slave-qfk4"
server_key: "b1b02d31b27586d4f76e60be1e7f1262"
server_key_iv: "875f22d84bc3b469f93bba236ea36b9a"
server_key_version: "encryptionkey@0"
I20260619 06:39:39.647737 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "8304899539bf4d5e87578ff990a95864"
format_stamp: "Formatted at 2026-06-19 06:39:39 on dist-test-slave-qfk4"
server_key: "b1b02d31b27586d4f76e60be1e7f1262"
server_key_iv: "875f22d84bc3b469f93bba236ea36b9a"
server_key_version: "encryptionkey@0"
I20260619 06:39:39.653045 8297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 06:39:39.656595 8312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:39.657974 8297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:39.658133 8297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "8304899539bf4d5e87578ff990a95864"
format_stamp: "Formatted at 2026-06-19 06:39:39 on dist-test-slave-qfk4"
server_key: "b1b02d31b27586d4f76e60be1e7f1262"
server_key_iv: "875f22d84bc3b469f93bba236ea36b9a"
server_key_version: "encryptionkey@0"
I20260619 06:39:39.658350 8297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:39.675873 8297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:39.680027 8297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:39.680295 8297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:39.697506 8297 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:44155
I20260619 06:39:39.697515 8363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:44155 every 8 connection(s)
I20260619 06:39:39.698863 8297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:39.702548 8364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:39.702809 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 8297
I20260619 06:39:39.703037 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:39:39.703635 7755 external_mini_cluster.cc:1428] Setting key 9b9a071b985facfedd444a9434553848
I20260619 06:39:39.716457 8364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864: Bootstrap starting.
Jun 19 06:39:39 dist-test-slave-qfk4 krb5kdc[8284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:39.720036 8364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:39.721174 8364 log.cc:824] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:39.724421 8364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864: No bootstrap required, opened a new log
I20260619 06:39:39.727418 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:39.705603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.7.146.254:44155)
0619 06:39:39.706391 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:39.706417 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:39.706468 (+ 51us) server_negotiation.cc:407] Connection header received
0619 06:39:39.708925 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:39.708954 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:39.709491 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:39.709959 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:39.711191 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:39.713132 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:39.714388 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:39.715425 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:39.715602 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:39.719929 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:39.719941 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:39.719957 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:39.719996 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:39.723387 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:39.724094 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:39.724102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:39.724111 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:39.724308 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:39.724655 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:39.724660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:39.724664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:39.725190 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:39.725497 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:39.725994 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 06:39:39.726403 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":313,"threads_started":1}
I20260619 06:39:39.732188 8364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8304899539bf4d5e87578ff990a95864" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44155 } }
I20260619 06:39:39.732543 8364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:39.732681 8364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8304899539bf4d5e87578ff990a95864, State: Initialized, Role: FOLLOWER
I20260619 06:39:39.733280 8364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8304899539bf4d5e87578ff990a95864" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44155 } }
I20260619 06:39:39.733520 8364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:39.733651 8364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:39.733803 8364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:39.735810 8364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8304899539bf4d5e87578ff990a95864" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44155 } }
I20260619 06:39:39.736258 8364 leader_election.cc:302] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8304899539bf4d5e87578ff990a95864; no voters:
I20260619 06:39:39.736799 8364 leader_election.cc:288] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:39.736907 8369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:39.737514 8369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [term 1 LEADER]: Becoming Leader. State: Replica: 8304899539bf4d5e87578ff990a95864, State: Running, Role: LEADER
I20260619 06:39:39.738039 8369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8304899539bf4d5e87578ff990a95864" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44155 } }
I20260619 06:39:39.739001 8364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:39.740756 8371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8304899539bf4d5e87578ff990a95864. Latest consensus state: current_term: 1 leader_uuid: "8304899539bf4d5e87578ff990a95864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8304899539bf4d5e87578ff990a95864" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44155 } } }
I20260619 06:39:39.741199 8371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:39.740830 8370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8304899539bf4d5e87578ff990a95864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8304899539bf4d5e87578ff990a95864" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44155 } } }
I20260619 06:39:39.741343 8370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:39.742848 8376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:39.748047 8376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:39.756909 8376 catalog_manager.cc:1347] Generated new cluster ID: 0dac21aeba0140b18a8bd375a4f13c51
I20260619 06:39:39.757048 8376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:39.782831 8376 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:39.784204 8376 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:39.799284 8376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8304899539bf4d5e87578ff990a95864: Generated new TSK 0
I20260619 06:39:39.800235 8376 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:39:39.844555 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44155
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:40.186827 8390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:40.187119 8390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:40.187239 8390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:40.194664 8390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:40.194782 8390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:40.194928 8390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:39:40.204878 8390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44155
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:40.206133 8390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:40.207451 8390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:40.220263 8395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:40.220291 8396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:40.220314 8398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:40.221220 8390 server_base.cc:1034] running on GCE node
I20260619 06:39:40.221803 8390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:40.222743 8390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:40.223937 8390 hybrid_clock.cc:648] HybridClock initialized: now 1781851180223883 us; error 49 us; skew 500 ppm
I20260619 06:39:40.224246 8390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:40 dist-test-slave-qfk4 krb5kdc[8284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:40.227520 8390 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:39:40.228992 8390 webserver.cc:460] Webserver started at http://127.7.146.193:43443/ using document root <none> and password file <none>
I20260619 06:39:40.229761 8390 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:40.229858 8390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:40.230108 8390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:40.233007 8390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "8abb77548a53486cae0fc92ed56f4087"
format_stamp: "Formatted at 2026-06-19 06:39:40 on dist-test-slave-qfk4"
server_key: "7860113179327893a2360dbe41b376c6"
server_key_iv: "89a982ad4895cc572412cb45ae444924"
server_key_version: "encryptionkey@0"
I20260619 06:39:40.233743 8390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "8abb77548a53486cae0fc92ed56f4087"
format_stamp: "Formatted at 2026-06-19 06:39:40 on dist-test-slave-qfk4"
server_key: "7860113179327893a2360dbe41b376c6"
server_key_iv: "89a982ad4895cc572412cb45ae444924"
server_key_version: "encryptionkey@0"
I20260619 06:39:40.238884 8390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 06:39:40.242460 8405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:40.243913 8390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:39:40.244047 8390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "8abb77548a53486cae0fc92ed56f4087"
format_stamp: "Formatted at 2026-06-19 06:39:40 on dist-test-slave-qfk4"
server_key: "7860113179327893a2360dbe41b376c6"
server_key_iv: "89a982ad4895cc572412cb45ae444924"
server_key_version: "encryptionkey@0"
I20260619 06:39:40.244240 8390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:40.260910 8390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:40.265158 8390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:40.265445 8390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:40.266414 8390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:39:40.267791 8390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:40.267874 8390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:40.267997 8390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:40.268047 8390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:40.296202 8390 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:46243
I20260619 06:39:40.296255 8517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:46243 every 8 connection(s)
I20260619 06:39:40.297516 8390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:39:40.302793 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 8390
I20260619 06:39:40.303002 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:39:40.303757 7755 external_mini_cluster.cc:1428] Setting key 524a3b1b531852b9881c27946b995cec
Jun 19 06:39:40 dist-test-slave-qfk4 krb5kdc[8284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:40.319515 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:40.300263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:46059 (local address 127.7.146.254:44155)
0619 06:39:40.300707 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:40.300712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:40.301824 (+ 1112us) server_negotiation.cc:407] Connection header received
0619 06:39:40.304164 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:40.304169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:40.304240 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:40.304342 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:40.306103 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:40.307009 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:40.309245 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:40.309711 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:40.309803 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:40.313561 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:40.313566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:40.313569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:40.313599 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:40.316048 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:40.316998 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:40.317002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:40.317005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:40.317117 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:40.317614 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:40.317618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:40.317620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:40.317902 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:40.318012 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:40.318929 (+ 917us) server_negotiation.cc:299] Negotiation successful
0619 06:39:40.319169 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
I20260619 06:39:40.320871 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:40.300830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:44155 (local address 127.7.146.193:46059)
0619 06:39:40.301590 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:40.301638 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:40.303849 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:40.304607 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:40.304619 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:40.305218 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:40.305944 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:40.305963 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:40.307155 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:40.307160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:40.309112 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:40.309127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:40.309931 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:40.309938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:40.310196 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:40.310871 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:40.310898 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:40.313321 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:40.316205 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:40.316215 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:40.316234 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:40.316877 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:40.317215 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:40.317220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:40.317223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:40.317485 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:40.318020 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:40.318029 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:40.318493 (+ 464us) client_negotiation.cc:769] Sending connection context
0619 06:39:40.318797 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 06:39:40.319391 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":257,"threads_started":1}
I20260619 06:39:40.322544 8518 heartbeater.cc:347] Connected to a master server at 127.7.146.254:44155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
I20260619 06:39:40.322918 8518 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:40.323851 8518 heartbeater.cc:511] Master 127.7.146.254:44155 requested a full tablet report, sending...
I20260619 06:39:40.326310 8329 ts_manager.cc:194] Registered new tserver with Master: 8abb77548a53486cae0fc92ed56f4087 (127.7.146.193:46243)
I20260619 06:39:40.327723 8329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:46059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:39:40.343477 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44155
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:40.646843 8524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:40.647130 8524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:40.647259 8524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:40.654383 8524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:40.654521 8524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:40.654670 8524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:39:40.664510 8524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44155
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:40.665809 8524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:40.667176 8524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:40.679613 8529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:40.679656 8532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:40.679688 8530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:40.680624 8524 server_base.cc:1034] running on GCE node
I20260619 06:39:40.681237 8524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:40.682235 8524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:40.683429 8524 hybrid_clock.cc:648] HybridClock initialized: now 1781851180683372 us; error 55 us; skew 500 ppm
I20260619 06:39:40.683732 8524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:40 dist-test-slave-qfk4 krb5kdc[8284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:40.686785 8524 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:39:40.688408 8524 webserver.cc:460] Webserver started at http://127.7.146.194:34075/ using document root <none> and password file <none>
I20260619 06:39:40.689144 8524 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:40.689261 8524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:40.689561 8524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:40.692381 8524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef"
format_stamp: "Formatted at 2026-06-19 06:39:40 on dist-test-slave-qfk4"
server_key: "5446b40774f0be2f193986b9648028e0"
server_key_iv: "ccef466972b668cda87558e5f31a70ca"
server_key_version: "encryptionkey@0"
I20260619 06:39:40.693202 8524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef"
format_stamp: "Formatted at 2026-06-19 06:39:40 on dist-test-slave-qfk4"
server_key: "5446b40774f0be2f193986b9648028e0"
server_key_iv: "ccef466972b668cda87558e5f31a70ca"
server_key_version: "encryptionkey@0"
I20260619 06:39:40.698302 8524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:39:40.701797 8539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:40.703321 8524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:39:40.703485 8524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef"
format_stamp: "Formatted at 2026-06-19 06:39:40 on dist-test-slave-qfk4"
server_key: "5446b40774f0be2f193986b9648028e0"
server_key_iv: "ccef466972b668cda87558e5f31a70ca"
server_key_version: "encryptionkey@0"
I20260619 06:39:40.703680 8524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:40.733867 8524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:40.738566 8524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:40.738869 8524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:40.739957 8524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:39:40.741338 8524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:40.741408 8524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:40.741504 8524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:40.741556 8524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:40.773485 8524 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:46733
I20260619 06:39:40.773557 8651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:46733 every 8 connection(s)
I20260619 06:39:40.774741 8524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:39:40.778765 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 8524
I20260619 06:39:40.778961 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:39:40.779520 7755 external_mini_cluster.cc:1428] Setting key 7e6c9e2d5eda94053313ac934eaa02ca
Jun 19 06:39:40 dist-test-slave-qfk4 krb5kdc[8284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851180, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
I20260619 06:39:40.795879 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:40.777336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:39789 (local address 127.7.146.254:44155)
0619 06:39:40.777515 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:40.777520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:40.778642 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 06:39:40.781050 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:40.781056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:40.781158 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:40.781242 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:40.783276 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:40.784014 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:40.786222 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:40.786647 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:40.786761 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:40.790194 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:40.790199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:40.790203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:40.790240 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:40.792902 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:40.793635 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:40.793639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:40.793641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:40.793780 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:40.794192 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:40.794195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:40.794197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:40.794448 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:40.794578 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:40.795315 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 06:39:40.795531 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:39:40.797158 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:40.777721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:44155 (local address 127.7.146.194:39789)
0619 06:39:40.778437 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:40.778499 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:40.780744 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:40.781495 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:40.781508 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:40.782260 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:40.783072 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:40.783100 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:40.784165 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:40.784170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:40.786056 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:40.786070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:40.786870 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:40.786878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:40.787096 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:40.787747 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:40.787770 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:40.789948 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:40.793054 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:40.793064 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:40.793082 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:40.793504 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:40.793886 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:40.793891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:40.793894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:40.794073 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:40.794565 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:40.794573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:40.794973 (+ 400us) client_negotiation.cc:769] Sending connection context
0619 06:39:40.795209 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 06:39:40.795749 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":252,"threads_started":1}
I20260619 06:39:40.798632 8652 heartbeater.cc:347] Connected to a master server at 127.7.146.254:44155
I20260619 06:39:40.798931 8652 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:40.799723 8652 heartbeater.cc:511] Master 127.7.146.254:44155 requested a full tablet report, sending...
I20260619 06:39:40.801331 8329 ts_manager.cc:194] Registered new tserver with Master: 1afc43fc5f0b413c8fa49ac3abc0a1ef (127.7.146.194:46733)
I20260619 06:39:40.802394 8329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:39789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:39:40.815852 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44155
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:41.123150 8657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:41.123446 8657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:41.123579 8657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:41.130761 8657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:41.130895 8657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:41.131042 8657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:39:41.140869 8657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44155
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:41.142177 8657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:41.143507 8657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:41.156142 8662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:41.156159 8663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:41.156435 8665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:41.157145 8657 server_base.cc:1034] running on GCE node
I20260619 06:39:41.157745 8657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:41.158679 8657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:41.159890 8657 hybrid_clock.cc:648] HybridClock initialized: now 1781851181159835 us; error 43 us; skew 500 ppm
I20260619 06:39:41.160202 8657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:41 dist-test-slave-qfk4 krb5kdc[8284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851181, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:41.163504 8657 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:39:41.165087 8657 webserver.cc:460] Webserver started at http://127.7.146.195:34055/ using document root <none> and password file <none>
I20260619 06:39:41.165776 8657 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:41.165889 8657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:41.166169 8657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:41.169106 8657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "2b6196cb1feb480692f30692f56e27f9"
format_stamp: "Formatted at 2026-06-19 06:39:41 on dist-test-slave-qfk4"
server_key: "a87745e7e12a575831d82bb395d029fb"
server_key_iv: "2ef4a849167d3c5ad8a7ae837f401ce7"
server_key_version: "encryptionkey@0"
I20260619 06:39:41.169885 8657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "2b6196cb1feb480692f30692f56e27f9"
format_stamp: "Formatted at 2026-06-19 06:39:41 on dist-test-slave-qfk4"
server_key: "a87745e7e12a575831d82bb395d029fb"
server_key_iv: "2ef4a849167d3c5ad8a7ae837f401ce7"
server_key_version: "encryptionkey@0"
I20260619 06:39:41.174929 8657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 06:39:41.178365 8672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:41.179757 8657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:39:41.179909 8657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "2b6196cb1feb480692f30692f56e27f9"
format_stamp: "Formatted at 2026-06-19 06:39:41 on dist-test-slave-qfk4"
server_key: "a87745e7e12a575831d82bb395d029fb"
server_key_iv: "2ef4a849167d3c5ad8a7ae837f401ce7"
server_key_version: "encryptionkey@0"
I20260619 06:39:41.180089 8657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:41.196712 8657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:41.201136 8657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:41.201403 8657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:41.202360 8657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:39:41.203687 8657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:41.203763 8657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:41.203879 8657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:41.203929 8657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:41.233239 8657 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:41203
I20260619 06:39:41.233282 8784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:41203 every 8 connection(s)
I20260619 06:39:41.234484 8657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:39:41.240816 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 8657
I20260619 06:39:41.241010 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:39:41.241511 7755 external_mini_cluster.cc:1428] Setting key 825d6fcdcb007d721bf20199bffa03d1
Jun 19 06:39:41 dist-test-slave-qfk4 krb5kdc[8284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851181, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:41.254130 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.236838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:39047 (local address 127.7.146.254:44155)
0619 06:39:41.237007 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.237011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.238181 (+ 1170us) server_negotiation.cc:407] Connection header received
0619 06:39:41.239929 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.239935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.240020 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.240104 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:41.242097 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.243080 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.244942 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.245391 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.245502 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.248903 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:41.248907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:41.248911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:41.248944 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:41.251164 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.251878 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.251882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.251884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.252007 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.252434 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.252438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.252441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.252821 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:41.252936 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.253637 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.253876 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:39:41.255118 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.237371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:44155 (local address 127.7.146.195:39047)
0619 06:39:41.238027 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:41.238064 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:41.239656 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:41.240361 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:41.240372 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:41.241099 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:41.241881 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.241906 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.243275 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.243279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.244816 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.244828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.245569 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.245573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.245762 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.246411 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:41.246433 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:41.248662 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:41.251312 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:41.251319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:41.251331 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:41.251740 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:41.252091 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:41.252094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:41.252096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:41.252278 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:41.252940 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:41.252946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:41.253277 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 06:39:41.253514 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 06:39:41.253917 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":253,"threads_started":1}
I20260619 06:39:41.256704 8785 heartbeater.cc:347] Connected to a master server at 127.7.146.254:44155
I20260619 06:39:41.257014 8785 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:41.257700 8785 heartbeater.cc:511] Master 127.7.146.254:44155 requested a full tablet report, sending...
I20260619 06:39:41.258999 8329 ts_manager.cc:194] Registered new tserver with Master: 2b6196cb1feb480692f30692f56e27f9 (127.7.146.195:41203)
I20260619 06:39:41.259761 8329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:39047
I20260619 06:39:41.269126 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:39:41.287285 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.275210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.7.146.254:44155)
0619 06:39:41.275376 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.275381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.275837 (+ 456us) server_negotiation.cc:407] Connection header received
0619 06:39:41.275989 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.275992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.276055 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.276128 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:41.277621 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.278312 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.279716 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.280134 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.280225 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.282446 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:41.282451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:41.282455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:41.282486 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:41.285007 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.285580 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.285584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.285586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.285707 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.286096 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.286099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.286101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.286406 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:41.286580 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.286903 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.287061 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:39:41.293704 8329 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:39:41.300457 8329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:39:41.322368 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.314477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:46243 (local address 127.0.0.1:41230)
0619 06:39:41.314962 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:41.314979 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:41.315086 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:41.316089 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:41.316094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:41.316117 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:41.316488 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.316501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.318253 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.318257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.319993 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.320009 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.321383 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.321390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.321569 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.321617 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 06:39:41.321739 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 06:39:41.321921 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":2,"thread_start_us":573,"threads_started":1}
I20260619 06:39:41.322556 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.312160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:41203 (local address 127.0.0.1:56816)
0619 06:39:41.312890 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:41.312931 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:41.313113 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:41.315237 (+ 2124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:41.315241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:41.315282 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:41.315919 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.315933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.317816 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.317824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.320026 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.320039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.321772 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.321778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.321950 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.321977 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:39:41.322069 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:39:41.322190 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":108,"spinlock_wait_cycles":384,"thread_start_us":1487,"threads_started":1}
I20260619 06:39:41.323043 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.314917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.7.146.193:46243)
0619 06:39:41.315582 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.315589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.315613 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:39:41.315697 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.315704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.315907 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.316701 (+ 794us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:41.316901 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.318136 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.320160 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.321209 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.321328 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.321565 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.322605 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.322768 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":379,"threads_started":1}
I20260619 06:39:41.323050 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.312978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.7.146.194:46733)
0619 06:39:41.313759 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.313769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.314231 (+ 462us) server_negotiation.cc:407] Connection header received
0619 06:39:41.314527 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.314535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.314782 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.314945 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:41.315982 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.317305 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.319924 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.321757 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.321878 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.322036 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.322571 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.322728 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":507,"threads_started":1}
I20260619 06:39:41.323050 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.312437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.7.146.195:41203)
0619 06:39:41.314720 (+ 2283us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.314727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.314769 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:39:41.314860 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.314867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.315066 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.315516 (+ 450us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:41.316118 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.317682 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.320191 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.321540 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.321697 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.321911 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.322123 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.322290 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2138,"thread_start_us":1987,"threads_started":1}
I20260619 06:39:41.323764 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.313311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:46733 (local address 127.0.0.1:60100)
0619 06:39:41.314136 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:41.314152 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:41.314339 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:41.315233 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:41.315241 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:41.315291 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:41.315793 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.315806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.317423 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.317428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.319720 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.319733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.322178 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.322184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.322357 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.322388 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:39:41.323277 (+ 889us) client_negotiation.cc:240] Negotiation successful
0619 06:39:41.323405 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":31,"spinlock_wait_cycles":5376,"thread_start_us":588,"threads_started":1}
I20260619 06:39:41.326640 8587 tablet_service.cc:1467] Processing CreateTablet for tablet 44366581a4604db59f8bd755a9f409cf (DEFAULT_TABLE table=test-table [id=d702983d70264a609424d0c6917b5074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:41.326720 8720 tablet_service.cc:1467] Processing CreateTablet for tablet 44366581a4604db59f8bd755a9f409cf (DEFAULT_TABLE table=test-table [id=d702983d70264a609424d0c6917b5074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:41.326720 8453 tablet_service.cc:1467] Processing CreateTablet for tablet 44366581a4604db59f8bd755a9f409cf (DEFAULT_TABLE table=test-table [id=d702983d70264a609424d0c6917b5074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:41.327818 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44366581a4604db59f8bd755a9f409cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:41.328002 8453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44366581a4604db59f8bd755a9f409cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:41.328015 8720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44366581a4604db59f8bd755a9f409cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:41.330698 8518 heartbeater.cc:503] Master 127.7.146.254:44155 was elected leader, sending a full tablet report...
I20260619 06:39:41.340055 8803 tablet_bootstrap.cc:492] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef: Bootstrap starting.
I20260619 06:39:41.344063 8805 tablet_bootstrap.cc:492] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9: Bootstrap starting.
I20260619 06:39:41.344374 8803 tablet_bootstrap.cc:654] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:41.343318 8804 tablet_bootstrap.cc:492] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087: Bootstrap starting.
I20260619 06:39:41.345813 8803 log.cc:824] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:41.347299 8805 tablet_bootstrap.cc:654] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:41.347482 8804 tablet_bootstrap.cc:654] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:41.348511 8805 log.cc:824] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:41.348529 8804 log.cc:824] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:41.349155 8803 tablet_bootstrap.cc:492] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef: No bootstrap required, opened a new log
I20260619 06:39:41.349474 8803 ts_tablet_manager.cc:1397] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260619 06:39:41.351485 8804 tablet_bootstrap.cc:492] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087: No bootstrap required, opened a new log
I20260619 06:39:41.351513 8805 tablet_bootstrap.cc:492] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9: No bootstrap required, opened a new log
I20260619 06:39:41.351737 8804 ts_tablet_manager.cc:1397] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:39:41.351743 8805 ts_tablet_manager.cc:1397] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 06:39:41.358603 8805 raft_consensus.cc:348] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6196cb1feb480692f30692f56e27f9" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41203 } } peers { permanent_uuid: "8abb77548a53486cae0fc92ed56f4087" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46243 } } peers { permanent_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46733 } }
I20260619 06:39:41.358966 8805 raft_consensus.cc:374] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:41.359077 8805 raft_consensus.cc:729] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6196cb1feb480692f30692f56e27f9, State: Initialized, Role: FOLLOWER
I20260619 06:39:41.359486 8803 raft_consensus.cc:348] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6196cb1feb480692f30692f56e27f9" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41203 } } peers { permanent_uuid: "8abb77548a53486cae0fc92ed56f4087" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46243 } } peers { permanent_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46733 } }
I20260619 06:39:41.359658 8805 consensus_queue.cc:260] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6196cb1feb480692f30692f56e27f9" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41203 } } peers { permanent_uuid: "8abb77548a53486cae0fc92ed56f4087" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46243 } } peers { permanent_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46733 } }
I20260619 06:39:41.359892 8803 raft_consensus.cc:374] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:41.360010 8803 raft_consensus.cc:729] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1afc43fc5f0b413c8fa49ac3abc0a1ef, State: Initialized, Role: FOLLOWER
I20260619 06:39:41.360824 8785 heartbeater.cc:503] Master 127.7.146.254:44155 was elected leader, sending a full tablet report...
I20260619 06:39:41.360721 8803 consensus_queue.cc:260] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6196cb1feb480692f30692f56e27f9" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41203 } } peers { permanent_uuid: "8abb77548a53486cae0fc92ed56f4087" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46243 } } peers { permanent_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46733 } }
I20260619 06:39:41.361526 8805 ts_tablet_manager.cc:1428] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9: Time spent starting tablet: real 0.010s user 0.009s sys 0.002s
I20260619 06:39:41.361676 8804 raft_consensus.cc:348] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6196cb1feb480692f30692f56e27f9" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41203 } } peers { permanent_uuid: "8abb77548a53486cae0fc92ed56f4087" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46243 } } peers { permanent_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46733 } }
I20260619 06:39:41.362071 8804 raft_consensus.cc:374] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:41.362161 8804 raft_consensus.cc:729] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abb77548a53486cae0fc92ed56f4087, State: Initialized, Role: FOLLOWER
I20260619 06:39:41.362421 8652 heartbeater.cc:503] Master 127.7.146.254:44155 was elected leader, sending a full tablet report...
I20260619 06:39:41.362771 8804 consensus_queue.cc:260] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6196cb1feb480692f30692f56e27f9" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41203 } } peers { permanent_uuid: "8abb77548a53486cae0fc92ed56f4087" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46243 } } peers { permanent_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46733 } }
I20260619 06:39:41.365271 8804 ts_tablet_manager.cc:1428] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 06:39:41.365330 8803 ts_tablet_manager.cc:1428] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20260619 06:39:41.405227 8809 raft_consensus.cc:482] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:39:41.405494 8809 raft_consensus.cc:504] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6196cb1feb480692f30692f56e27f9" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41203 } } peers { permanent_uuid: "8abb77548a53486cae0fc92ed56f4087" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46243 } } peers { permanent_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46733 } }
I20260619 06:39:41.406186 8811 raft_consensus.cc:482] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:39:41.406495 8811 raft_consensus.cc:504] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6196cb1feb480692f30692f56e27f9" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41203 } } peers { permanent_uuid: "8abb77548a53486cae0fc92ed56f4087" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46243 } } peers { permanent_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46733 } }
I20260619 06:39:41.407301 8809 leader_election.cc:288] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8abb77548a53486cae0fc92ed56f4087 (127.7.146.193:46243), 1afc43fc5f0b413c8fa49ac3abc0a1ef (127.7.146.194:46733)
I20260619 06:39:41.415156 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.407739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:46243 (local address 127.7.146.195:56083)
0619 06:39:41.408097 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:41.408114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:41.408241 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:41.408848 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:41.408853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:41.408893 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:41.409074 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.409085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.410537 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.410542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.412515 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.412531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.414346 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.414353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.414553 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.414586 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:39:41.414671 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:39:41.414814 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260619 06:39:41.415803 8811 leader_election.cc:288] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b6196cb1feb480692f30692f56e27f9 (127.7.146.195:41203), 1afc43fc5f0b413c8fa49ac3abc0a1ef (127.7.146.194:46733)
I20260619 06:39:41.416234 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.407846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:56083 (local address 127.7.146.193:46243)
0619 06:39:41.408059 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.408065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.408360 (+ 295us) server_negotiation.cc:407] Connection header received
0619 06:39:41.408432 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.408438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.408513 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.408678 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:41.409232 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.410385 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.412716 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.414017 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.414169 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.414246 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.415821 (+ 1575us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.415973 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:39:41.418324 8473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44366581a4604db59f8bd755a9f409cf" candidate_uuid: "2b6196cb1feb480692f30692f56e27f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abb77548a53486cae0fc92ed56f4087" is_pre_election: true
I20260619 06:39:41.418877 8473 raft_consensus.cc:2454] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b6196cb1feb480692f30692f56e27f9 in term 0.
I20260619 06:39:41.419750 8675 leader_election.cc:302] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b6196cb1feb480692f30692f56e27f9, 8abb77548a53486cae0fc92ed56f4087; no voters:
I20260619 06:39:41.420217 8809 raft_consensus.cc:2781] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:39:41.420344 8809 raft_consensus.cc:482] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:39:41.420454 8809 raft_consensus.cc:3037] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:41.422653 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.408145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:38777 (local address 127.7.146.194:46733)
0619 06:39:41.408417 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.408423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.415510 (+ 7087us) server_negotiation.cc:407] Connection header received
0619 06:39:41.415670 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.415676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.415755 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.415843 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:41.416669 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.417682 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.420267 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.421447 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.421563 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.421635 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.422286 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.422413 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260619 06:39:41.423533 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.407757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:46733 (local address 127.7.146.195:38777)
0619 06:39:41.415414 (+ 7657us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:41.415435 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:41.415547 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:41.416083 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:41.416089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:41.416121 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:41.416457 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.416470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.417872 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.417877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.420080 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.420095 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.421894 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.421900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.422100 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.422132 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:39:41.423017 (+ 885us) client_negotiation.cc:240] Negotiation successful
0619 06:39:41.423158 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7548,"mutex_wait_us":887,"thread_start_us":9224,"threads_started":1}
I20260619 06:39:41.423631 8809 raft_consensus.cc:504] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6196cb1feb480692f30692f56e27f9" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41203 } } peers { permanent_uuid: "8abb77548a53486cae0fc92ed56f4087" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46243 } } peers { permanent_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46733 } }
I20260619 06:39:41.425038 8809 leader_election.cc:288] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [CANDIDATE]: Term 1 election: Requested vote from peers 8abb77548a53486cae0fc92ed56f4087 (127.7.146.193:46243), 1afc43fc5f0b413c8fa49ac3abc0a1ef (127.7.146.194:46733)
I20260619 06:39:41.424844 8607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44366581a4604db59f8bd755a9f409cf" candidate_uuid: "2b6196cb1feb480692f30692f56e27f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" is_pre_election: true
I20260619 06:39:41.425309 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.418165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:46893 (local address 127.7.146.195:41203)
0619 06:39:41.418346 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.418352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.418370 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:39:41.418425 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.418429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.418490 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.418570 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:41.419322 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.420485 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.422946 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.424127 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.424303 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.424378 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.424972 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.425098 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:39:41.425485 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.408687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:39159 (local address 127.7.146.194:46733)
0619 06:39:41.416865 (+ 8178us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.416871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.417781 (+ 910us) server_negotiation.cc:407] Connection header received
0619 06:39:41.418002 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.418007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.418071 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.418131 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:41.419157 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.420336 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.422989 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.424212 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.424374 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.424440 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.424999 (+ 559us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.425132 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8043,"thread_start_us":9121,"threads_started":1}
I20260619 06:39:41.425448 8607 raft_consensus.cc:2454] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b6196cb1feb480692f30692f56e27f9 in term 0.
I20260619 06:39:41.425755 8606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44366581a4604db59f8bd755a9f409cf" candidate_uuid: "2b6196cb1feb480692f30692f56e27f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef"
I20260619 06:39:41.425993 8606 raft_consensus.cc:3037] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:41.426048 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.408360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:41203 (local address 127.7.146.193:46893)
0619 06:39:41.417309 (+ 8949us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:41.417337 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:41.417470 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:41.418708 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:41.418712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:41.418736 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:41.419163 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.419175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.420690 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.420695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.422784 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.422798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.424530 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.424536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.424743 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.424821 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 06:39:41.425451 (+ 630us) client_negotiation.cc:240] Negotiation successful
0619 06:39:41.425673 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8819,"mutex_wait_us":10,"thread_start_us":9025,"threads_started":1}
I20260619 06:39:41.426208 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.408732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:46733 (local address 127.7.146.193:39159)
0619 06:39:41.417695 (+ 8963us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:41.417712 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:41.417871 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:41.418406 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:41.418411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:41.418440 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:41.418929 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.418946 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.420504 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.420510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.422823 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:41.422836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.424596 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.424603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:41.424746 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.424852 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 06:39:41.425644 (+ 792us) client_negotiation.cc:240] Negotiation successful
0619 06:39:41.425796 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8846,"mutex_wait_us":247,"thread_start_us":8293,"threads_started":1}
I20260619 06:39:41.426429 8473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44366581a4604db59f8bd755a9f409cf" candidate_uuid: "2b6196cb1feb480692f30692f56e27f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abb77548a53486cae0fc92ed56f4087"
I20260619 06:39:41.426635 8473 raft_consensus.cc:3037] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:41.427457 8607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44366581a4604db59f8bd755a9f409cf" candidate_uuid: "8abb77548a53486cae0fc92ed56f4087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" is_pre_election: true
I20260619 06:39:41.427618 8740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44366581a4604db59f8bd755a9f409cf" candidate_uuid: "8abb77548a53486cae0fc92ed56f4087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b6196cb1feb480692f30692f56e27f9" is_pre_election: true
I20260619 06:39:41.428225 8740 raft_consensus.cc:2379] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8abb77548a53486cae0fc92ed56f4087 in current term 1: Already voted for candidate 2b6196cb1feb480692f30692f56e27f9 in this term.
I20260619 06:39:41.429011 8408 leader_election.cc:302] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8abb77548a53486cae0fc92ed56f4087; no voters: 1afc43fc5f0b413c8fa49ac3abc0a1ef, 2b6196cb1feb480692f30692f56e27f9
I20260619 06:39:41.429311 8606 raft_consensus.cc:2454] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b6196cb1feb480692f30692f56e27f9 in term 1.
I20260619 06:39:41.429666 8473 raft_consensus.cc:2454] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b6196cb1feb480692f30692f56e27f9 in term 1.
I20260619 06:39:41.429698 8674 leader_election.cc:302] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1afc43fc5f0b413c8fa49ac3abc0a1ef, 2b6196cb1feb480692f30692f56e27f9; no voters:
I20260619 06:39:41.429859 8811 raft_consensus.cc:2726] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 06:39:41.430014 8809 raft_consensus.cc:2781] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:41.430462 8809 raft_consensus.cc:686] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [term 1 LEADER]: Becoming Leader. State: Replica: 2b6196cb1feb480692f30692f56e27f9, State: Running, Role: LEADER
I20260619 06:39:41.430900 8809 consensus_queue.cc:237] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6196cb1feb480692f30692f56e27f9" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41203 } } peers { permanent_uuid: "8abb77548a53486cae0fc92ed56f4087" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46243 } } peers { permanent_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46733 } }
I20260619 06:39:41.435444 8329 catalog_manager.cc:5495] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b6196cb1feb480692f30692f56e27f9 (127.7.146.195). New cstate: current_term: 1 leader_uuid: "2b6196cb1feb480692f30692f56e27f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6196cb1feb480692f30692f56e27f9" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8abb77548a53486cae0fc92ed56f4087" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1afc43fc5f0b413c8fa49ac3abc0a1ef" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46733 } health_report { overall_health: UNKNOWN } } }
Jun 19 06:39:41 dist-test-slave-qfk4 krb5kdc[8284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:39:41.483064 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.468611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.7.146.193:46243)
0619 06:39:41.468821 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.468828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.469374 (+ 546us) server_negotiation.cc:407] Connection header received
0619 06:39:41.469508 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.469512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.469579 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.469675 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:41.471258 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.472090 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.473352 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.473804 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.473921 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.477630 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:41.477645 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:41.477661 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:41.477696 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:41.480400 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.481108 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.481116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.481119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.481252 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.481601 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.481606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.481608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.481962 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:41.482244 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.482519 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.482734 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 06:39:41.486049 8786 tablet.cc:2369] T 44366581a4604db59f8bd755a9f409cf P 2b6196cb1feb480692f30692f56e27f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:39:41 dist-test-slave-qfk4 krb5kdc[8284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:39:41 dist-test-slave-qfk4 krb5kdc[8284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:39:41.524343 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.510837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.7.146.193:46243)
0619 06:39:41.511030 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.511036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.511382 (+ 346us) server_negotiation.cc:407] Connection header received
0619 06:39:41.511550 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.511555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.511653 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.511750 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:41.513190 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.513935 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.515147 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.515552 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.515660 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.519383 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:41.519388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:41.519391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:41.519426 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:41.521904 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.522514 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.522519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.522521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.522635 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.523005 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.523010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.523014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.523292 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:41.523546 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.523826 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.524026 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:39:41.526571 8653 tablet.cc:2369] T 44366581a4604db59f8bd755a9f409cf P 1afc43fc5f0b413c8fa49ac3abc0a1ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:39:41 dist-test-slave-qfk4 krb5kdc[8284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260619 06:39:41.549819 8519 tablet.cc:2369] T 44366581a4604db59f8bd755a9f409cf P 8abb77548a53486cae0fc92ed56f4087: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:39:41 dist-test-slave-qfk4 krb5kdc[8284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:39:41.565434 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.552039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.7.146.193:46243)
0619 06:39:41.552241 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.552248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.552543 (+ 295us) server_negotiation.cc:407] Connection header received
0619 06:39:41.552808 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.552812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.552878 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.552961 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:41.554268 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.554954 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.556150 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.556606 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.556717 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.560298 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:41.560305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:41.560309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:41.560342 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:41.562859 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.563479 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.563483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.563486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.563606 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.563975 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.563987 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.563989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.564302 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:41.564535 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.564856 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.565082 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 06:39:41.567009 8433 tablet_service.cc:2908] Rejecting scan request for tablet 44366581a4604db59f8bd755a9f409cf: Uninitialized: clean time has not yet been initialized
I20260619 06:39:41.569720 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 8390
I20260619 06:39:41.581429 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.570586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.7.146.193:46243)
0619 06:39:41.570786 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.570792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.570932 (+ 140us) server_negotiation.cc:407] Connection header received
0619 06:39:41.571048 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.571051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.571119 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.571194 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:41.572239 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.573060 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.574227 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.574635 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.574776 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.576750 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:41.576755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:41.576758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:41.576789 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:41.579027 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.579631 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.579636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.579638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.579802 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.580065 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.580068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.580070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.580324 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:41.580558 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.580869 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.581104 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:39:41.582098 8513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:41.671015 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 8390
I20260619 06:39:41.684028 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 8524
Jun 19 06:39:41 dist-test-slave-qfk4 krb5kdc[8284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:39:41.698539 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.684817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.7.146.194:46733)
0619 06:39:41.685008 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.685012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.685029 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:39:41.685199 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.685203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.685271 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.685344 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:41.686484 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.687175 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.688403 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.688933 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.689032 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.692894 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:41.692909 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:41.692925 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:41.692960 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:41.695695 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.696343 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.696350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.696352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.696488 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.696880 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.696883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.696885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.697348 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:41.697643 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.697945 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.698187 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:39:41.699270 8647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:41.791150 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 8524
I20260619 06:39:41.803915 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 8657
Jun 19 06:39:41 dist-test-slave-qfk4 krb5kdc[8284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:39:41.817409 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:41.804664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.7.146.195:41203)
0619 06:39:41.804830 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:41.804834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:41.804852 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:39:41.804984 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:41.804987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:41.805050 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:41.805138 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:41.806279 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.807057 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.808343 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:41.808837 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:41.808927 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:41.812643 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:41.812657 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:41.812673 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:41.812711 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:41.815207 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.815753 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.815760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.815762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.815882 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:41.816143 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:41.816146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:41.816148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:41.816484 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:41.816837 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:41.816946 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 06:39:41.817091 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:39:41.818066 8780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:41.904779 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 8657
I20260619 06:39:41.917114 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 8297
I20260619 06:39:41.918200 8359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:42.024479 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 8297
2026-06-19T06:39:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3897 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:42 dist-test-slave-qfk4 krb5kdc[8835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:42 dist-test-slave-qfk4 krb5kdc[8835](info): set up 2 sockets
Jun 19 06:39:42 dist-test-slave-qfk4 krb5kdc[8835](info): commencing operation
krb5kdc: starting...
W20260619 06:39:43.016470 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:39:43 dist-test-slave-qfk4 krb5kdc[8835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:39:43.182154 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:33073
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:33073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:43.492444 8848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:43.492786 8848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:43.492901 8848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:43.500243 8848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:43.500355 8848 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:39:43.500418 8848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:43.500510 8848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:43.500556 8848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:43.500672 8848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:43.511003 8848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:33073
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:33073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:43.512283 8848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:43.513624 8848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:43.524979 8853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:43.525071 8854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:43.525545 8856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:43.525822 8848 server_base.cc:1034] running on GCE node
I20260619 06:39:43.526620 8848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:43.528035 8848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:43.529268 8848 hybrid_clock.cc:648] HybridClock initialized: now 1781851183529203 us; error 34 us; skew 500 ppm
I20260619 06:39:43.529608 8848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:43 dist-test-slave-qfk4 krb5kdc[8835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:43.532713 8848 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:39:43.534179 8848 webserver.cc:460] Webserver started at http://127.7.146.254:36621/ using document root <none> and password file <none>
I20260619 06:39:43.534843 8848 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:43.534924 8848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:43.535183 8848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:43.538121 8848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "ef9a5232821c4bcfa54049fa928626e1"
format_stamp: "Formatted at 2026-06-19 06:39:43 on dist-test-slave-qfk4"
server_key: "1c810d05678d632313fc889fedd26968"
server_key_iv: "11e9c6f4710abd63f5155858938401b8"
server_key_version: "encryptionkey@0"
I20260619 06:39:43.538874 8848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "ef9a5232821c4bcfa54049fa928626e1"
format_stamp: "Formatted at 2026-06-19 06:39:43 on dist-test-slave-qfk4"
server_key: "1c810d05678d632313fc889fedd26968"
server_key_iv: "11e9c6f4710abd63f5155858938401b8"
server_key_version: "encryptionkey@0"
I20260619 06:39:43.543951 8848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 06:39:43.547377 8863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:43.548780 8848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:39:43.548944 8848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "ef9a5232821c4bcfa54049fa928626e1"
format_stamp: "Formatted at 2026-06-19 06:39:43 on dist-test-slave-qfk4"
server_key: "1c810d05678d632313fc889fedd26968"
server_key_iv: "11e9c6f4710abd63f5155858938401b8"
server_key_version: "encryptionkey@0"
I20260619 06:39:43.549146 8848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:43.566640 8848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:43.570673 8848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:43.570919 8848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:43.590942 8848 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:33073
I20260619 06:39:43.590961 8924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:33073 every 8 connection(s)
I20260619 06:39:43.592311 8848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:43.596110 8925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:43.598634 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 8848
I20260619 06:39:43.598886 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:39:43.599489 7755 external_mini_cluster.cc:1428] Setting key 36ab272f4da7490939d6a2b5c7f84342
I20260619 06:39:43.608606 8925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1: Bootstrap starting.
I20260619 06:39:43.612043 8925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:43.613036 8925 log.cc:824] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:39:43 dist-test-slave-qfk4 krb5kdc[8835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:43.615769 8925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1: No bootstrap required, opened a new log
I20260619 06:39:43.622063 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:43.601550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.7.146.254:33073)
0619 06:39:43.602306 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:43.602321 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:43.602374 (+ 53us) server_negotiation.cc:407] Connection header received
0619 06:39:43.604494 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:43.604524 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:43.605089 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:43.605569 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:43.606926 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:43.608810 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:43.610057 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:43.611043 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:43.611226 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:43.614991 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:43.615004 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:43.615022 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:43.615062 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:43.618193 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:43.618843 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:43.618849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:43.618855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:43.619015 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:43.619362 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:43.619367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:43.619371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:43.619908 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:43.620190 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:43.620636 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 06:39:43.621028 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":304,"threads_started":1}
I20260619 06:39:43.623399 8925 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9a5232821c4bcfa54049fa928626e1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33073 } }
I20260619 06:39:43.623823 8925 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:43.623925 8925 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9a5232821c4bcfa54049fa928626e1, State: Initialized, Role: FOLLOWER
I20260619 06:39:43.624480 8925 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9a5232821c4bcfa54049fa928626e1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33073 } }
I20260619 06:39:43.624769 8925 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:43.624895 8925 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:43.625025 8925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:43.626988 8925 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9a5232821c4bcfa54049fa928626e1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33073 } }
I20260619 06:39:43.627408 8925 leader_election.cc:302] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9a5232821c4bcfa54049fa928626e1; no voters:
I20260619 06:39:43.627939 8925 leader_election.cc:288] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:43.628062 8930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:43.629002 8930 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [term 1 LEADER]: Becoming Leader. State: Replica: ef9a5232821c4bcfa54049fa928626e1, State: Running, Role: LEADER
I20260619 06:39:43.629452 8930 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9a5232821c4bcfa54049fa928626e1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33073 } }
I20260619 06:39:43.629964 8925 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:43.632194 8932 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9a5232821c4bcfa54049fa928626e1. Latest consensus state: current_term: 1 leader_uuid: "ef9a5232821c4bcfa54049fa928626e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9a5232821c4bcfa54049fa928626e1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33073 } } }
I20260619 06:39:43.632577 8932 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:43.632225 8931 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef9a5232821c4bcfa54049fa928626e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9a5232821c4bcfa54049fa928626e1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33073 } } }
I20260619 06:39:43.633721 8931 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:43.634960 8938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:43.639796 8938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:43.648327 8938 catalog_manager.cc:1347] Generated new cluster ID: 81b9847bcc0c41dfbd46ed5779b6c37d
I20260619 06:39:43.648433 8938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:43.659355 8938 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:43.660670 8938 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:43.667671 8938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1: Generated new TSK 0
I20260619 06:39:43.668745 8938 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:39:43.716537 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33073
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:44.029554 8951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:44.029898 8951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:44.030046 8951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:44.037386 8951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:44.037510 8951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:44.037634 8951 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:39:44.037823 8951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:39:44.048815 8951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.146.254:33073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:44.050112 8951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:44.051441 8951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:44.063915 8957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:44.063954 8959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:44.063989 8956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:44.064544 8951 server_base.cc:1034] running on GCE node
I20260619 06:39:44.065186 8951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:44.066160 8951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:44.067380 8951 hybrid_clock.cc:648] HybridClock initialized: now 1781851184067315 us; error 42 us; skew 500 ppm
I20260619 06:39:44.067656 8951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:44 dist-test-slave-qfk4 krb5kdc[8835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851184, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:44.070694 8951 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:39:44.072111 8951 webserver.cc:460] Webserver started at http://127.7.146.193:33375/ using document root <none> and password file <none>
I20260619 06:39:44.072837 8951 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:44.072952 8951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:44.073244 8951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:44.076020 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "fd871d765e074093b19093eeb27fdc6e"
format_stamp: "Formatted at 2026-06-19 06:39:44 on dist-test-slave-qfk4"
server_key: "d472a9c3a1355ddb2b424d51172d96ee"
server_key_iv: "0d64ff5984545180597736b47e6d174d"
server_key_version: "encryptionkey@0"
I20260619 06:39:44.076848 8951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "fd871d765e074093b19093eeb27fdc6e"
format_stamp: "Formatted at 2026-06-19 06:39:44 on dist-test-slave-qfk4"
server_key: "d472a9c3a1355ddb2b424d51172d96ee"
server_key_iv: "0d64ff5984545180597736b47e6d174d"
server_key_version: "encryptionkey@0"
I20260619 06:39:44.081900 8951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:39:44.085263 8966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:44.086611 8951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:44.086738 8951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "fd871d765e074093b19093eeb27fdc6e"
format_stamp: "Formatted at 2026-06-19 06:39:44 on dist-test-slave-qfk4"
server_key: "d472a9c3a1355ddb2b424d51172d96ee"
server_key_iv: "0d64ff5984545180597736b47e6d174d"
server_key_version: "encryptionkey@0"
I20260619 06:39:44.086938 8951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:44.106557 8951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:44.110952 8951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:44.111207 8951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:44.113965 8951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:44.114068 8951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:44.114280 8951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:44.114358 8951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:39:44 dist-test-slave-qfk4 krb5kdc[8835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851184, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:44.138921 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:44.115355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33073 (local address 127.7.146.193:59069)
0619 06:39:44.116200 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:44.116242 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:44.118202 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:44.119048 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:44.119064 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:44.119976 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:44.121053 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:44.121086 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.122579 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.122583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:44.124417 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:44.124433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.125505 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.125512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:44.125852 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:44.126672 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:44.126717 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:44.130035 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:44.133831 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:44.133844 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:44.133866 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:44.134496 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:44.135080 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:44.135083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:44.135089 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:44.135310 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:44.135854 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:44.135862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:44.136342 (+ 480us) client_negotiation.cc:769] Sending connection context
0619 06:39:44.136707 (+ 365us) client_negotiation.cc:240] Negotiation successful
0619 06:39:44.137352 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":376,"threads_started":1}
I20260619 06:39:44.139756 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:44.114630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:59069 (local address 127.7.146.254:33073)
0619 06:39:44.114820 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:44.114826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:44.116365 (+ 1539us) server_negotiation.cc:407] Connection header received
0619 06:39:44.118507 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:44.118513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:44.118604 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:44.118711 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:44.121309 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.122423 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.124641 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.125208 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.125332 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:44.130397 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:44.130404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:44.130410 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:44.130451 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:44.133646 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:44.134618 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:44.134624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:44.134627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:44.134945 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:44.135435 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:44.135441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:44.135445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:44.135735 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:44.135867 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:44.139284 (+ 3417us) server_negotiation.cc:299] Negotiation successful
0619 06:39:44.139462 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:39:44.158695 8951 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:41247
I20260619 06:39:44.158715 9085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:41247 every 8 connection(s)
I20260619 06:39:44.159152 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:44.145983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:47367 (local address 127.7.146.254:33073)
0619 06:39:44.146129 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:44.146134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:44.146748 (+ 614us) server_negotiation.cc:407] Connection header received
0619 06:39:44.146917 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:44.146923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:44.146987 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:44.147064 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:44.147979 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.148753 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.150549 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.150988 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.151124 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:44.153416 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:44.153422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:44.153428 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:44.153468 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:44.156425 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:44.157244 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:44.157250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:44.157254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:44.157381 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:44.157712 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:44.157717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:44.157720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:44.158070 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:44.158206 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:44.158600 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 06:39:44.158816 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:39:44.159307 9055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:44.146051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33073 (local address 127.7.146.193:47367)
0619 06:39:44.146627 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:44.146643 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:44.146797 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:44.147101 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:44.147105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:44.147536 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:44.147839 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:44.147850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.148879 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.148885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:44.150407 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:44.150420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.151215 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.151223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:44.151436 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:44.152088 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:44.152133 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:44.153238 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:44.156539 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:44.156544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:44.156548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:44.157126 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:44.157479 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:44.157484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:44.157488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:44.157615 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:44.158182 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:44.158187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:44.158377 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 06:39:44.158504 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 06:39:44.158845 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":311,"threads_started":1}
I20260619 06:39:44.160210 8951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:39:44.162309 9086 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33073
I20260619 06:39:44.162536 9086 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:44.163281 9086 heartbeater.cc:511] Master 127.7.146.254:33073 requested a full tablet report, sending...
I20260619 06:39:44.164099 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 8951
I20260619 06:39:44.164314 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:39:44.164985 7755 external_mini_cluster.cc:1428] Setting key fe5883e98b1f77f10168677b3d07bcc4
I20260619 06:39:44.165581 8880 ts_manager.cc:194] Registered new tserver with Master: fd871d765e074093b19093eeb27fdc6e (127.7.146.193:41247)
I20260619 06:39:44.167356 8880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:59069
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:39:44.205394 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33073
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:44.515836 9090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:44.516135 9090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:44.516273 9090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:44.523639 9090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:44.523764 9090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:44.523890 9090 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:39:44.523998 9090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:39:44.533970 9090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.146.254:33073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:44.535295 9090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:44.536682 9090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:44.549156 9095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:44.549398 9096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:44.549424 9098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:44.550314 9090 server_base.cc:1034] running on GCE node
I20260619 06:39:44.550936 9090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:44.551831 9090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:44.553033 9090 hybrid_clock.cc:648] HybridClock initialized: now 1781851184552962 us; error 49 us; skew 500 ppm
I20260619 06:39:44.553314 9090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:44 dist-test-slave-qfk4 krb5kdc[8835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851184, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:44.556288 9090 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:39:44.557730 9090 webserver.cc:460] Webserver started at http://127.7.146.194:45075/ using document root <none> and password file <none>
I20260619 06:39:44.558388 9090 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:44.558486 9090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:44.558765 9090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:44.561615 9090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "9dfccfcc939d459ea922377b61078b13"
format_stamp: "Formatted at 2026-06-19 06:39:44 on dist-test-slave-qfk4"
server_key: "38006209471adb4ec0600d33725be639"
server_key_iv: "6d382bd9594b6fabe28bf4e536f677ad"
server_key_version: "encryptionkey@0"
I20260619 06:39:44.562340 9090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "9dfccfcc939d459ea922377b61078b13"
format_stamp: "Formatted at 2026-06-19 06:39:44 on dist-test-slave-qfk4"
server_key: "38006209471adb4ec0600d33725be639"
server_key_iv: "6d382bd9594b6fabe28bf4e536f677ad"
server_key_version: "encryptionkey@0"
I20260619 06:39:44.567373 9090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:39:44.570744 9105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:44.572046 9090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 06:39:44.572186 9090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "9dfccfcc939d459ea922377b61078b13"
format_stamp: "Formatted at 2026-06-19 06:39:44 on dist-test-slave-qfk4"
server_key: "38006209471adb4ec0600d33725be639"
server_key_iv: "6d382bd9594b6fabe28bf4e536f677ad"
server_key_version: "encryptionkey@0"
I20260619 06:39:44.572384 9090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:44.586524 9090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:44.590598 9090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:44.590842 9090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:44.593811 9090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:44.593896 9090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:44.594013 9090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:44.594096 9090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:39:44 dist-test-slave-qfk4 krb5kdc[8835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851184, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:44.619109 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:44.595670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:44529 (local address 127.7.146.254:33073)
0619 06:39:44.595845 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:44.595851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:44.596181 (+ 330us) server_negotiation.cc:407] Connection header received
0619 06:39:44.599086 (+ 2905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:44.599093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:44.599185 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:44.599263 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:44.601753 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.602923 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.605745 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.606447 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.606559 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:44.611683 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:44.611688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:44.611692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:44.611726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:44.615038 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:44.616059 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:44.616064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:44.616068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:44.616213 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:44.616878 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:44.616883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:44.616886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:44.617198 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:44.617320 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:44.618448 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0619 06:39:44.618696 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:39:44.621066 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:44.595111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33073 (local address 127.7.146.194:44529)
0619 06:39:44.596009 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:44.596059 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:44.598692 (+ 2633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:44.599782 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:44.599794 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:44.600597 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:44.601558 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:44.601589 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.603141 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.603146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:44.605548 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:44.605565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.606874 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.606882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:44.607180 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:44.607958 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:44.607995 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:44.611413 (+ 3418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:44.615256 (+ 3843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:44.615267 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:44.615291 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:44.615914 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:44.616338 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:44.616344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:44.616347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:44.616738 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:44.617307 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:44.617317 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:44.617942 (+ 625us) client_negotiation.cc:769] Sending connection context
0619 06:39:44.618332 (+ 390us) client_negotiation.cc:240] Negotiation successful
0619 06:39:44.619134 (+ 802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":352,"threads_started":1}
I20260619 06:39:44.631536 9090 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:35967
I20260619 06:39:44.631536 9224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:35967 every 8 connection(s)
I20260619 06:39:44.632879 9090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:39:44.634796 9225 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33073
I20260619 06:39:44.635022 9225 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:44.635665 9225 heartbeater.cc:511] Master 127.7.146.254:33073 requested a full tablet report, sending...
I20260619 06:39:44.636981 8880 ts_manager.cc:194] Registered new tserver with Master: 9dfccfcc939d459ea922377b61078b13 (127.7.146.194:35967)
I20260619 06:39:44.637939 8880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:44529
I20260619 06:39:44.641062 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:44.627964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:43453 (local address 127.7.146.254:33073)
0619 06:39:44.628148 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:44.628154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:44.628973 (+ 819us) server_negotiation.cc:407] Connection header received
0619 06:39:44.629168 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:44.629174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:44.629258 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:44.629362 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:44.630575 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.631563 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.632933 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.633542 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.633668 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:44.635679 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:44.635684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:44.635688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:44.635717 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:44.638522 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:44.639281 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:44.639284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:44.639287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:44.639410 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:44.639759 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:44.639761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:44.639763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:44.640027 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:44.640131 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:44.640603 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 06:39:44.640808 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:39:44.641150 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:44.628109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33073 (local address 127.7.146.194:43453)
0619 06:39:44.628836 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:44.628871 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:44.629031 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:44.629413 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:44.629419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:44.630032 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:44.630425 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:44.630440 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.631736 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.631741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:44.632801 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:44.632813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:44.633932 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:44.633940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:44.634136 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:44.634695 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:44.634712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:44.635509 (+ 797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:44.638642 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:44.638647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:44.638650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:44.639181 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:44.639509 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:44.639514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:44.639518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:44.639664 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:44.640131 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:44.640136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:44.640305 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 06:39:44.640475 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 06:39:44.640780 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":389,"threads_started":1}
I20260619 06:39:44.642370 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 9090
I20260619 06:39:44.642565 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:39:44.643095 7755 external_mini_cluster.cc:1428] Setting key 122a48236d30f164ea4a27195871cc13
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:39:44.679382 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33073
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:44.986717 9229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:44.987015 9229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:44.987144 9229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:44.994287 9229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:44.994407 9229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:44.994534 9229 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:39:44.994644 9229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:39:45.004204 9229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.146.254:33073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:45.005497 9229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:45.006851 9229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:45.019656 9234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:45.019608 9235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:45.019902 9237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:45.020504 9229 server_base.cc:1034] running on GCE node
I20260619 06:39:45.021134 9229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:45.022070 9229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:45.023240 9229 hybrid_clock.cc:648] HybridClock initialized: now 1781851185023173 us; error 46 us; skew 500 ppm
I20260619 06:39:45.023522 9229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:45 dist-test-slave-qfk4 krb5kdc[8835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851185, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:45.026574 9229 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:39:45.028004 9229 webserver.cc:460] Webserver started at http://127.7.146.195:42591/ using document root <none> and password file <none>
I20260619 06:39:45.028712 9229 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:45.028813 9229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:45.029081 9229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:45.031991 9229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "671c78dfb25742c69bdd432e1430ca01"
format_stamp: "Formatted at 2026-06-19 06:39:45 on dist-test-slave-qfk4"
server_key: "85eabbf14ea31e7380c78a232de68f59"
server_key_iv: "00c7bd04b129a953832fc5b37d636ef9"
server_key_version: "encryptionkey@0"
I20260619 06:39:45.032829 9229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "671c78dfb25742c69bdd432e1430ca01"
format_stamp: "Formatted at 2026-06-19 06:39:45 on dist-test-slave-qfk4"
server_key: "85eabbf14ea31e7380c78a232de68f59"
server_key_iv: "00c7bd04b129a953832fc5b37d636ef9"
server_key_version: "encryptionkey@0"
I20260619 06:39:45.038391 9229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:39:45.042333 9244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:45.043936 9229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:39:45.044097 9229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "671c78dfb25742c69bdd432e1430ca01"
format_stamp: "Formatted at 2026-06-19 06:39:45 on dist-test-slave-qfk4"
server_key: "85eabbf14ea31e7380c78a232de68f59"
server_key_iv: "00c7bd04b129a953832fc5b37d636ef9"
server_key_version: "encryptionkey@0"
I20260619 06:39:45.044309 9229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:45.062443 9229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:45.066490 9229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:45.066738 9229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:45.069324 9229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:45.069406 9229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:45.069556 9229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:45.069612 9229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:39:45 dist-test-slave-qfk4 krb5kdc[8835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851185, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:45.092607 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.069895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:51867 (local address 127.7.146.254:33073)
0619 06:39:45.070089 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.070095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.071624 (+ 1529us) server_negotiation.cc:407] Connection header received
0619 06:39:45.074171 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.074177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.074279 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.074348 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:45.076770 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.077776 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.079995 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.080584 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.080728 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.085570 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:45.085576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:45.085580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:45.085623 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:45.089181 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:45.089936 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:45.089942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:45.089946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:45.090151 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:45.090634 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:45.090639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:45.090642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:45.091010 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:45.091168 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.091935 (+ 767us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.092193 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:39:45.093603 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.070565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33073 (local address 127.7.146.195:51867)
0619 06:39:45.071419 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:45.071476 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:45.073772 (+ 2296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:45.074860 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:45.074868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:45.075601 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:45.076516 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.076544 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.077922 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.077929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.079839 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.079857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.081018 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.081026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.081316 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.082133 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:45.082167 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:45.085322 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:45.089348 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:45.089357 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:45.089371 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:45.089806 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:45.090281 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:45.090286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:45.090290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:45.090512 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:45.091155 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:45.091164 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:45.091571 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 06:39:45.091807 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 06:39:45.092226 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":421,"threads_started":1}
I20260619 06:39:45.114320 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.099425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33073 (local address 127.7.146.195:40541)
0619 06:39:45.099969 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:45.099984 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:45.100113 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:45.100429 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:45.100433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:45.101046 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:45.101525 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.101544 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.103089 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.103095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.104996 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.105011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.105974 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.105981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.106163 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.106896 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:45.106918 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:45.107975 (+ 1057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:45.111449 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:45.111455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:45.111459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:45.112048 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:45.112434 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:45.112439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:45.112446 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:45.112649 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:45.113365 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:45.113369 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:45.113540 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 06:39:45.113649 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 06:39:45.113924 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":295,"threads_started":1}
I20260619 06:39:45.114406 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.099363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:40541 (local address 127.7.146.254:33073)
0619 06:39:45.099507 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.099511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.100067 (+ 556us) server_negotiation.cc:407] Connection header received
0619 06:39:45.100237 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.100242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.100303 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.100380 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:45.101706 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.102877 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.105141 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.105740 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.105870 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.108182 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:45.108187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:45.108190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:45.108237 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:45.111311 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:45.112164 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:45.112170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:45.112174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:45.112329 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:45.112822 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:45.112827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:45.112830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:45.113156 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:45.113310 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.113837 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.114045 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:39:45.117761 9229 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:43601
I20260619 06:39:45.117827 9363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:43601 every 8 connection(s)
I20260619 06:39:45.118923 9229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:39:45.120987 9364 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33073
I20260619 06:39:45.121188 9364 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:45.121798 9364 heartbeater.cc:511] Master 127.7.146.254:33073 requested a full tablet report, sending...
I20260619 06:39:45.123162 8880 ts_manager.cc:194] Registered new tserver with Master: 671c78dfb25742c69bdd432e1430ca01 (127.7.146.195:43601)
I20260619 06:39:45.124038 8880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:51867
I20260619 06:39:45.124941 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 9229
I20260619 06:39:45.125105 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:39:45.125509 7755 external_mini_cluster.cc:1428] Setting key afc091db64893459aaeda00907cca573
I20260619 06:39:45.128283 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:39:45 dist-test-slave-qfk4 krb5kdc[8835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851185, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:39:45 dist-test-slave-qfk4 krb5kdc[8835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851185, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:45.170615 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.157258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.7.146.254:33073)
0619 06:39:45.157497 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.157504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.157765 (+ 261us) server_negotiation.cc:407] Connection header received
0619 06:39:45.157960 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.157964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.158040 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.158143 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:45.159532 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.160303 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.161609 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.162032 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.162169 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.165770 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:45.165774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:45.165778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:45.165812 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:45.168178 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:45.168841 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:45.168845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:45.168847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:45.168973 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:45.169313 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:45.169316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:45.169318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:45.169583 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:45.169799 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.170126 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.170331 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:39:45.170843 9086 heartbeater.cc:503] Master 127.7.146.254:33073 was elected leader, sending a full tablet report...
I20260619 06:39:45.174961 8879 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:39:45.181759 8879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:39:45.220005 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.193233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.7.146.194:35967)
0619 06:39:45.209070 (+ 15837us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.209081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.213151 (+ 4070us) server_negotiation.cc:407] Connection header received
0619 06:39:45.213283 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.213291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.213585 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.214507 (+ 922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:45.214721 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.215916 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.218087 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.218942 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.219064 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.219277 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.219573 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.219734 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15638,"thread_start_us":15487,"threads_started":1}
I20260619 06:39:45.220095 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.193065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:35967 (local address 127.0.0.1:52898)
0619 06:39:45.212230 (+ 19165us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:45.212284 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:45.212582 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:45.213846 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:45.213854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:45.213887 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:45.214314 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.214326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.216037 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.216042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.217904 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.217921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.219128 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.219133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.219349 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.219391 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 06:39:45.219517 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 06:39:45.219704 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18871,"mutex_wait_us":8,"thread_start_us":20419,"threads_started":1}
I20260619 06:39:45.222098 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.193339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:41247 (local address 127.0.0.1:43654)
0619 06:39:45.213136 (+ 19797us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:45.213154 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:45.213268 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:45.214771 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:45.214774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:45.214794 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:45.215204 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.215216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.217236 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.217242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.219519 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.219532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.221345 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.221351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.221521 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.221555 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:39:45.221620 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 06:39:45.221753 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19691,"mutex_wait_us":34,"thread_start_us":19560,"threads_started":1}
I20260619 06:39:45.222738 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.193922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.7.146.195:43601)
0619 06:39:45.210927 (+ 17005us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.210937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.213836 (+ 2899us) server_negotiation.cc:407] Connection header received
0619 06:39:45.214107 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.214114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.214348 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.214503 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:45.215498 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.217112 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.220031 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.221582 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.221665 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.221868 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.222291 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.222447 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16794,"thread_start_us":17008,"threads_started":1}
I20260619 06:39:45.223016 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.193550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.7.146.193:41247)
0619 06:39:45.211592 (+ 18042us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.211601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.213270 (+ 1669us) server_negotiation.cc:407] Connection header received
0619 06:39:45.213394 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.213402 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.214512 (+ 1110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.215361 (+ 849us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:45.215548 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.217088 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.219690 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.221122 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.222401 (+ 1279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.222536 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.222582 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.222755 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17874,"thread_start_us":17740,"threads_started":1}
I20260619 06:39:45.223420 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.193601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:43601 (local address 127.0.0.1:39024)
0619 06:39:45.213734 (+ 20133us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:45.213751 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:45.213920 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:45.214808 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:45.214813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:45.214841 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:45.215278 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.215292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.217265 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.217647 (+ 382us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.219866 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.219878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.221898 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.221906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.222083 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.222114 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:39:45.222957 (+ 843us) client_negotiation.cc:240] Negotiation successful
0619 06:39:45.223082 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20006,"mutex_wait_us":89,"thread_start_us":18169,"threads_started":1}
I20260619 06:39:45.224378 9155 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb3b00eea8c4f44bb292fb98691b251 (DEFAULT_TABLE table=test-table [id=73b4c3396fd8460eb9d33ab00d53a980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:45.225829 9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb3b00eea8c4f44bb292fb98691b251. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:45.226540 9294 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb3b00eea8c4f44bb292fb98691b251 (DEFAULT_TABLE table=test-table [id=73b4c3396fd8460eb9d33ab00d53a980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:45.226466 9016 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb3b00eea8c4f44bb292fb98691b251 (DEFAULT_TABLE table=test-table [id=73b4c3396fd8460eb9d33ab00d53a980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:45.227768 9016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb3b00eea8c4f44bb292fb98691b251. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:45.227782 9294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb3b00eea8c4f44bb292fb98691b251. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:45.237493 9383 tablet_bootstrap.cc:492] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: Bootstrap starting.
I20260619 06:39:45.240502 9383 tablet_bootstrap.cc:654] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:45.241564 9383 log.cc:824] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:45.242919 9384 tablet_bootstrap.cc:492] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: Bootstrap starting.
I20260619 06:39:45.242434 9385 tablet_bootstrap.cc:492] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: Bootstrap starting.
I20260619 06:39:45.245494 9383 tablet_bootstrap.cc:492] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: No bootstrap required, opened a new log
I20260619 06:39:45.245738 9383 ts_tablet_manager.cc:1397] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 06:39:45.246532 9384 tablet_bootstrap.cc:654] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:45.246531 9385 tablet_bootstrap.cc:654] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:45.247725 9385 log.cc:824] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:45.247725 9384 log.cc:824] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:45.250913 9385 tablet_bootstrap.cc:492] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: No bootstrap required, opened a new log
I20260619 06:39:45.251001 9384 tablet_bootstrap.cc:492] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: No bootstrap required, opened a new log
I20260619 06:39:45.251152 9385 ts_tablet_manager.cc:1397] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 06:39:45.251259 9384 ts_tablet_manager.cc:1397] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 06:39:45.254966 9383 raft_consensus.cc:348] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.255409 9383 raft_consensus.cc:374] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:45.255514 9383 raft_consensus.cc:729] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfccfcc939d459ea922377b61078b13, State: Initialized, Role: FOLLOWER
I20260619 06:39:45.256140 9383 consensus_queue.cc:260] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.257304 9225 heartbeater.cc:503] Master 127.7.146.254:33073 was elected leader, sending a full tablet report...
I20260619 06:39:45.257920 9383 ts_tablet_manager.cc:1428] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260619 06:39:45.259606 9385 raft_consensus.cc:348] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.259939 9385 raft_consensus.cc:374] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:45.260035 9385 raft_consensus.cc:729] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671c78dfb25742c69bdd432e1430ca01, State: Initialized, Role: FOLLOWER
I20260619 06:39:45.260654 9385 consensus_queue.cc:260] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.260895 9384 raft_consensus.cc:348] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.261355 9384 raft_consensus.cc:374] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:45.261705 9364 heartbeater.cc:503] Master 127.7.146.254:33073 was elected leader, sending a full tablet report...
I20260619 06:39:45.262320 9385 ts_tablet_manager.cc:1428] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260619 06:39:45.261484 9384 raft_consensus.cc:729] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd871d765e074093b19093eeb27fdc6e, State: Initialized, Role: FOLLOWER
I20260619 06:39:45.263262 9384 consensus_queue.cc:260] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.265084 9384 ts_tablet_manager.cc:1428] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
W20260619 06:39:45.370452 9365 tablet.cc:2369] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:39:45.385205 9226 tablet.cc:2369] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:39:45.412776 9087 tablet.cc:2369] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:39:45.439149 9391 raft_consensus.cc:482] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:39:45.439476 9391 raft_consensus.cc:504] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.441248 9391 leader_election.cc:288] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dfccfcc939d459ea922377b61078b13 (127.7.146.194:35967), 671c78dfb25742c69bdd432e1430ca01 (127.7.146.195:43601)
I20260619 06:39:45.448022 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.441681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:47141 (local address 127.7.146.194:35967)
0619 06:39:45.441927 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.441934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.442838 (+ 904us) server_negotiation.cc:407] Connection header received
0619 06:39:45.443069 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.443075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.443144 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.443215 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:45.443854 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.444557 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.446363 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.447101 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.447256 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.447331 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.447645 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.447777 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 06:39:45.448029 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.441716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:34531 (local address 127.7.146.195:43601)
0619 06:39:45.441931 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.441936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.442849 (+ 913us) server_negotiation.cc:407] Connection header received
0619 06:39:45.443058 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.443064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.443132 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.443206 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:45.443831 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.444538 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.446364 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.447137 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.447282 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.447352 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.447667 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.447799 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:39:45.448176 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.441805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:35967 (local address 127.7.146.193:47141)
0619 06:39:45.442729 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:45.442744 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:45.442910 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:45.443247 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:45.443251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:45.443272 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:45.443649 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.443658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.444685 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.444689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.446211 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.446222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.447308 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.447314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.447450 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.447524 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 06:39:45.447600 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:39:45.447732 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":65,"spinlock_wait_cycles":15360,"thread_start_us":566,"threads_started":1}
I20260619 06:39:45.448210 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.441782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:43601 (local address 127.7.146.193:34531)
0619 06:39:45.442705 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:45.442728 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:45.442908 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:45.443231 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:45.443235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:45.443257 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:45.443699 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.443707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.444679 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.444684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.446207 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.446219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.447352 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.447358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.447529 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.447552 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:39:45.447628 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:39:45.447758 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":74,"spinlock_wait_cycles":29312,"thread_start_us":593,"threads_started":1}
I20260619 06:39:45.449373 9314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb3b00eea8c4f44bb292fb98691b251" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671c78dfb25742c69bdd432e1430ca01" is_pre_election: true
I20260619 06:39:45.449448 9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb3b00eea8c4f44bb292fb98691b251" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfccfcc939d459ea922377b61078b13" is_pre_election: true
I20260619 06:39:45.450011 9175 raft_consensus.cc:2454] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd871d765e074093b19093eeb27fdc6e in term 0.
I20260619 06:39:45.450037 9314 raft_consensus.cc:2454] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd871d765e074093b19093eeb27fdc6e in term 0.
I20260619 06:39:45.450810 8969 leader_election.cc:302] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671c78dfb25742c69bdd432e1430ca01, fd871d765e074093b19093eeb27fdc6e; no voters:
I20260619 06:39:45.451241 9391 raft_consensus.cc:2781] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:39:45.451421 9391 raft_consensus.cc:482] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:39:45.451507 9391 raft_consensus.cc:3037] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:45.453900 9391 raft_consensus.cc:504] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.454499 9391 leader_election.cc:288] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 election: Requested vote from peers 9dfccfcc939d459ea922377b61078b13 (127.7.146.194:35967), 671c78dfb25742c69bdd432e1430ca01 (127.7.146.195:43601)
I20260619 06:39:45.455052 9314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb3b00eea8c4f44bb292fb98691b251" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671c78dfb25742c69bdd432e1430ca01"
I20260619 06:39:45.455052 9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb3b00eea8c4f44bb292fb98691b251" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfccfcc939d459ea922377b61078b13"
I20260619 06:39:45.455289 9314 raft_consensus.cc:3037] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:45.455288 9175 raft_consensus.cc:3037] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:45.458428 9314 raft_consensus.cc:2454] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd871d765e074093b19093eeb27fdc6e in term 1.
I20260619 06:39:45.458428 9175 raft_consensus.cc:2454] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd871d765e074093b19093eeb27fdc6e in term 1.
I20260619 06:39:45.458849 8967 leader_election.cc:302] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 671c78dfb25742c69bdd432e1430ca01, 9dfccfcc939d459ea922377b61078b13, fd871d765e074093b19093eeb27fdc6e; no voters:
I20260619 06:39:45.459172 9391 raft_consensus.cc:2781] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:45.459564 9391 raft_consensus.cc:686] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 1 LEADER]: Becoming Leader. State: Replica: fd871d765e074093b19093eeb27fdc6e, State: Running, Role: LEADER
I20260619 06:39:45.460026 9391 consensus_queue.cc:237] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.464902 8880 catalog_manager.cc:5495] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e reported cstate change: term changed from 0 to 1, leader changed from <none> to fd871d765e074093b19093eeb27fdc6e (127.7.146.193). New cstate: current_term: 1 leader_uuid: "fd871d765e074093b19093eeb27fdc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } health_report { overall_health: UNKNOWN } } }
I20260619 06:39:45.525830 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.519884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.7.146.193:41247)
0619 06:39:45.520127 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.520134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.520189 (+ 55us) server_negotiation.cc:407] Connection header received
0619 06:39:45.520555 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.520559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.520692 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.520844 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:45.521242 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.521954 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.523337 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.523876 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.524007 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.524599 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:45.525156 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:45.525311 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.525438 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.525532 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 06:39:45.534425 9175 raft_consensus.cc:1261] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13 [term 1 FOLLOWER]: Refusing update from remote peer fd871d765e074093b19093eeb27fdc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:45.534425 9314 raft_consensus.cc:1261] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01 [term 1 FOLLOWER]: Refusing update from remote peer fd871d765e074093b19093eeb27fdc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:45.535614 9391 consensus_queue.cc:1035] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:45.535976 9394 consensus_queue.cc:1035] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:45.549381 9399 mvcc.cc:205] Tried to move back new op lower bound from 7298462455938043904 to 7298462455648415744. Current Snapshot: MvccSnapshot[applied={T|T < 7298462455938043904}]
I20260619 06:39:45.565539 8878 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:45.565953 8878 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:45.570251 8878 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1: Sending DeleteTablet for 3 replicas of tablet 6fb3b00eea8c4f44bb292fb98691b251
I20260619 06:39:45.571377 9294 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fb3b00eea8c4f44bb292fb98691b251 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:45 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:39024
I20260619 06:39:45.571492 9155 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fb3b00eea8c4f44bb292fb98691b251 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:45 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:52898
I20260619 06:39:45.572229 9016 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fb3b00eea8c4f44bb292fb98691b251 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:45 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:43654
I20260619 06:39:45.573227 8878 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:39:45.574126 8878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:39:45.576704 9408 tablet_replica.cc:330] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: stopping tablet replica
I20260619 06:39:45.577221 9408 raft_consensus.cc:2229] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:39:45.577811 9407 tablet_replica.cc:330] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: stopping tablet replica
I20260619 06:39:45.578145 9406 tablet_replica.cc:330] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: stopping tablet replica
I20260619 06:39:45.578507 9408 raft_consensus.cc:2258] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:45.578730 9406 raft_consensus.cc:2229] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:45.579136 9407 raft_consensus.cc:2229] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:45.579317 9406 raft_consensus.cc:2258] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:45.579633 9407 raft_consensus.cc:2258] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:45.583148 9407 ts_tablet_manager.cc:1905] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:45.583060 9016 tablet_service.cc:1467] Processing CreateTablet for tablet bc18edda99a44d9abea3a3192e8c4349 (DEFAULT_TABLE table=test-table [id=49d42efdddcc4dea91be80cbe4b3450a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:45.583549 9016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc18edda99a44d9abea3a3192e8c4349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:45.583563 9406 ts_tablet_manager.cc:1905] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:45.583878 9294 tablet_service.cc:1467] Processing CreateTablet for tablet bc18edda99a44d9abea3a3192e8c4349 (DEFAULT_TABLE table=test-table [id=49d42efdddcc4dea91be80cbe4b3450a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:45.584272 9294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc18edda99a44d9abea3a3192e8c4349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:45.589669 9155 tablet_service.cc:1467] Processing CreateTablet for tablet bc18edda99a44d9abea3a3192e8c4349 (DEFAULT_TABLE table=test-table [id=49d42efdddcc4dea91be80cbe4b3450a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:45.590541 9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc18edda99a44d9abea3a3192e8c4349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:45.591864 9384 tablet_bootstrap.cc:492] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e: Bootstrap starting.
I20260619 06:39:45.591879 9385 tablet_bootstrap.cc:492] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01: Bootstrap starting.
I20260619 06:39:45.583081 9408 ts_tablet_manager.cc:1905] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:45.593551 9385 tablet_bootstrap.cc:654] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:45.594138 9384 tablet_bootstrap.cc:654] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:45.599596 9408 ts_tablet_manager.cc:1918] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:39:45.599925 9408 log.cc:1196] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/6fb3b00eea8c4f44bb292fb98691b251
I20260619 06:39:45.600590 9408 ts_tablet_manager.cc:1939] T 6fb3b00eea8c4f44bb292fb98691b251 P fd871d765e074093b19093eeb27fdc6e: Deleting consensus metadata
I20260619 06:39:45.602480 8867 catalog_manager.cc:4841] TS fd871d765e074093b19093eeb27fdc6e (127.7.146.193:41247): tablet 6fb3b00eea8c4f44bb292fb98691b251 (table test-table [id=73b4c3396fd8460eb9d33ab00d53a980]) successfully deleted
I20260619 06:39:45.604291 9384 tablet_bootstrap.cc:492] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e: No bootstrap required, opened a new log
I20260619 06:39:45.604508 9384 ts_tablet_manager.cc:1397] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s
I20260619 06:39:45.606593 9384 raft_consensus.cc:348] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.607074 9384 raft_consensus.cc:374] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:45.607331 9384 raft_consensus.cc:729] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd871d765e074093b19093eeb27fdc6e, State: Initialized, Role: FOLLOWER
I20260619 06:39:45.607888 9384 consensus_queue.cc:260] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.609040 9384 ts_tablet_manager.cc:1428] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260619 06:39:45.610394 9407 ts_tablet_manager.cc:1918] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:39:45.610558 9407 log.cc:1196] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/6fb3b00eea8c4f44bb292fb98691b251
I20260619 06:39:45.611100 9407 ts_tablet_manager.cc:1939] T 6fb3b00eea8c4f44bb292fb98691b251 P 9dfccfcc939d459ea922377b61078b13: Deleting consensus metadata
I20260619 06:39:45.613610 9406 ts_tablet_manager.cc:1918] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:39:45.613783 9406 log.cc:1196] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/6fb3b00eea8c4f44bb292fb98691b251
I20260619 06:39:45.614305 9406 ts_tablet_manager.cc:1939] T 6fb3b00eea8c4f44bb292fb98691b251 P 671c78dfb25742c69bdd432e1430ca01: Deleting consensus metadata
I20260619 06:39:45.615981 8864 catalog_manager.cc:4841] TS 9dfccfcc939d459ea922377b61078b13 (127.7.146.194:35967): tablet 6fb3b00eea8c4f44bb292fb98691b251 (table test-table [id=73b4c3396fd8460eb9d33ab00d53a980]) successfully deleted
I20260619 06:39:45.616557 8866 catalog_manager.cc:4841] TS 671c78dfb25742c69bdd432e1430ca01 (127.7.146.195:43601): tablet 6fb3b00eea8c4f44bb292fb98691b251 (table test-table [id=73b4c3396fd8460eb9d33ab00d53a980]) successfully deleted
I20260619 06:39:45.616868 9383 tablet_bootstrap.cc:492] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13: Bootstrap starting.
I20260619 06:39:45.616987 9385 tablet_bootstrap.cc:492] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01: No bootstrap required, opened a new log
I20260619 06:39:45.617120 9385 ts_tablet_manager.cc:1397] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.006s
I20260619 06:39:45.617676 9385 raft_consensus.cc:348] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.617846 9385 raft_consensus.cc:374] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:45.617955 9385 raft_consensus.cc:729] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671c78dfb25742c69bdd432e1430ca01, State: Initialized, Role: FOLLOWER
I20260619 06:39:45.618170 9385 consensus_queue.cc:260] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.618587 9383 tablet_bootstrap.cc:654] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:45.618620 9385 ts_tablet_manager.cc:1428] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:39:45.625608 9383 tablet_bootstrap.cc:492] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13: No bootstrap required, opened a new log
I20260619 06:39:45.625788 9383 ts_tablet_manager.cc:1397] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 06:39:45.626614 9383 raft_consensus.cc:348] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.626875 9383 raft_consensus.cc:374] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:45.626995 9383 raft_consensus.cc:729] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfccfcc939d459ea922377b61078b13, State: Initialized, Role: FOLLOWER
I20260619 06:39:45.627301 9383 consensus_queue.cc:260] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.627951 9383 ts_tablet_manager.cc:1428] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 06:39:45.875334 9389 raft_consensus.cc:482] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:39:45.875558 9389 raft_consensus.cc:504] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.877373 9389 leader_election.cc:288] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd871d765e074093b19093eeb27fdc6e (127.7.146.193:41247), 671c78dfb25742c69bdd432e1430ca01 (127.7.146.195:43601)
I20260619 06:39:45.880661 9394 raft_consensus.cc:482] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:39:45.880890 9394 raft_consensus.cc:504] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.881599 9394 leader_election.cc:288] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dfccfcc939d459ea922377b61078b13 (127.7.146.194:35967), 671c78dfb25742c69bdd432e1430ca01 (127.7.146.195:43601)
I20260619 06:39:45.882175 9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc18edda99a44d9abea3a3192e8c4349" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfccfcc939d459ea922377b61078b13" is_pre_election: true
I20260619 06:39:45.882429 9175 raft_consensus.cc:2454] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd871d765e074093b19093eeb27fdc6e in term 0.
I20260619 06:39:45.882371 9314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc18edda99a44d9abea3a3192e8c4349" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671c78dfb25742c69bdd432e1430ca01" is_pre_election: true
I20260619 06:39:45.882674 9314 raft_consensus.cc:2454] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd871d765e074093b19093eeb27fdc6e in term 0.
I20260619 06:39:45.882874 8967 leader_election.cc:302] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dfccfcc939d459ea922377b61078b13, fd871d765e074093b19093eeb27fdc6e; no voters:
I20260619 06:39:45.883301 9394 raft_consensus.cc:2781] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:39:45.883423 9394 raft_consensus.cc:482] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:39:45.883533 9394 raft_consensus.cc:3037] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:45.885641 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.877854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:47747 (local address 127.7.146.193:41247)
0619 06:39:45.878050 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.878056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.878769 (+ 713us) server_negotiation.cc:407] Connection header received
0619 06:39:45.879066 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.879073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.879157 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.879260 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:45.879810 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.880846 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.883141 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.884378 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.884508 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.884607 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.885179 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.885332 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:39:45.886454 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.877794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:41247 (local address 127.7.146.194:47747)
0619 06:39:45.878656 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:45.878680 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:45.878898 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:45.879286 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:45.879292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:45.879319 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:45.879637 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.879650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.881009 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.881014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.882956 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.882971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.884799 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.884805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.884990 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.885023 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:39:45.885793 (+ 770us) client_negotiation.cc:240] Negotiation successful
0619 06:39:45.885924 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":462,"threads_started":1}
I20260619 06:39:45.887554 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.877867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:39365 (local address 127.7.146.195:43601)
0619 06:39:45.878049 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.878056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.879170 (+ 1114us) server_negotiation.cc:407] Connection header received
0619 06:39:45.879374 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.879379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.879454 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.879504 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:45.880213 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.881348 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.885421 (+ 4073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.886601 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.886721 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.886796 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.887181 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.887319 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:39:45.887521 9036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc18edda99a44d9abea3a3192e8c4349" candidate_uuid: "9dfccfcc939d459ea922377b61078b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd871d765e074093b19093eeb27fdc6e" is_pre_election: true
I20260619 06:39:45.887605 9394 raft_consensus.cc:504] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.888299 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.878420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:43601 (local address 127.7.146.194:39365)
0619 06:39:45.879051 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:45.879072 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:45.879201 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:45.879649 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:45.879652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:45.879669 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:45.880022 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.880033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.883279 (+ 3246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.883284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.885228 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.885254 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.886836 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.886843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.887010 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.887039 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:39:45.887822 (+ 783us) client_negotiation.cc:240] Negotiation successful
0619 06:39:45.887956 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":310,"threads_started":1}
I20260619 06:39:45.888685 9036 raft_consensus.cc:2379] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dfccfcc939d459ea922377b61078b13 in current term 1: Already voted for candidate fd871d765e074093b19093eeb27fdc6e in this term.
I20260619 06:39:45.888754 9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc18edda99a44d9abea3a3192e8c4349" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfccfcc939d459ea922377b61078b13"
I20260619 06:39:45.888933 9175 raft_consensus.cc:3037] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:45.889030 9394 leader_election.cc:288] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 election: Requested vote from peers 9dfccfcc939d459ea922377b61078b13 (127.7.146.194:35967), 671c78dfb25742c69bdd432e1430ca01 (127.7.146.195:43601)
I20260619 06:39:45.889304 9314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc18edda99a44d9abea3a3192e8c4349" candidate_uuid: "9dfccfcc939d459ea922377b61078b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671c78dfb25742c69bdd432e1430ca01" is_pre_election: true
I20260619 06:39:45.889525 9313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc18edda99a44d9abea3a3192e8c4349" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671c78dfb25742c69bdd432e1430ca01"
I20260619 06:39:45.889528 9314 raft_consensus.cc:2454] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dfccfcc939d459ea922377b61078b13 in term 0.
I20260619 06:39:45.890811 9108 leader_election.cc:302] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671c78dfb25742c69bdd432e1430ca01, 9dfccfcc939d459ea922377b61078b13; no voters:
I20260619 06:39:45.892099 9175 raft_consensus.cc:2454] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd871d765e074093b19093eeb27fdc6e in term 1.
I20260619 06:39:45.892289 9389 raft_consensus.cc:2741] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 06:39:45.892484 8967 leader_election.cc:302] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dfccfcc939d459ea922377b61078b13, fd871d765e074093b19093eeb27fdc6e; no voters: 671c78dfb25742c69bdd432e1430ca01
I20260619 06:39:45.892774 9394 raft_consensus.cc:2781] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:45.892931 9394 raft_consensus.cc:686] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 1 LEADER]: Becoming Leader. State: Replica: fd871d765e074093b19093eeb27fdc6e, State: Running, Role: LEADER
I20260619 06:39:45.893185 9394 consensus_queue.cc:237] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:45.895591 8880 catalog_manager.cc:5495] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e reported cstate change: term changed from 0 to 1, leader changed from <none> to fd871d765e074093b19093eeb27fdc6e (127.7.146.193). New cstate: current_term: 1 leader_uuid: "fd871d765e074093b19093eeb27fdc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } health_report { overall_health: UNKNOWN } } }
Jun 19 06:39:45 dist-test-slave-qfk4 krb5kdc[8835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851183, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:45.996542 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.982810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33073 (local address 127.0.0.1:45180)
0619 06:39:45.983630 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:45.983645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:45.983777 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:45.984113 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:45.984116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:45.984795 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:45.984972 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.984980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.986396 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.986400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.987322 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:45.987332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.988099 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.988104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:45.988234 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.988854 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:45.988870 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:45.991227 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:45.994060 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:45.994071 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:45.994078 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:45.994539 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:45.994885 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:45.994888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:45.994891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:45.995063 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:45.995664 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:45.995671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:45.995831 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 06:39:45.995978 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 06:39:45.996217 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"spinlock_wait_cycles":18304,"thread_start_us":563,"threads_started":1}
I20260619 06:39:45.996670 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:45.982826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.7.146.254:33073)
0619 06:39:45.983484 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:45.983490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:45.983725 (+ 235us) server_negotiation.cc:407] Connection header received
0619 06:39:45.983877 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:45.983882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:45.983955 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:45.984066 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:45.985100 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.986254 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.987448 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:45.987954 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:45.988057 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:45.991425 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:45.991429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:45.991433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:45.991459 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:45.993915 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:45.994638 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:45.994641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:45.994643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:45.994767 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:45.995198 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:45.995202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:45.995204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:45.995503 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:45.995638 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:45.996109 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 06:39:45.996325 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":479,"thread_start_us":326,"threads_started":1}
I20260619 06:39:46.001345 8880 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:45180:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:39:46.002465 8880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:39:46.009272 9016 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeaff653b2944008aef99473b84ff5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=271a616fd8a542a5846c4e387c5f09ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:39:46.009272 9294 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeaff653b2944008aef99473b84ff5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=271a616fd8a542a5846c4e387c5f09ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:39:46.009284 9155 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeaff653b2944008aef99473b84ff5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=271a616fd8a542a5846c4e387c5f09ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:39:46.009850 9294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeaff653b2944008aef99473b84ff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:46.009845 9016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeaff653b2944008aef99473b84ff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:46.009887 9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeaff653b2944008aef99473b84ff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:46.015625 9383 tablet_bootstrap.cc:492] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13: Bootstrap starting.
I20260619 06:39:46.016618 9385 tablet_bootstrap.cc:492] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01: Bootstrap starting.
I20260619 06:39:46.016619 9384 tablet_bootstrap.cc:492] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: Bootstrap starting.
I20260619 06:39:46.017282 9383 tablet_bootstrap.cc:654] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:46.018193 9384 tablet_bootstrap.cc:654] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:46.018199 9385 tablet_bootstrap.cc:654] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:46.019933 9383 tablet_bootstrap.cc:492] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13: No bootstrap required, opened a new log
I20260619 06:39:46.020097 9383 ts_tablet_manager.cc:1397] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:39:46.020853 9383 raft_consensus.cc:348] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:46.021051 9383 raft_consensus.cc:374] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:46.021126 9383 raft_consensus.cc:729] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfccfcc939d459ea922377b61078b13, State: Initialized, Role: FOLLOWER
I20260619 06:39:46.021184 9384 tablet_bootstrap.cc:492] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: No bootstrap required, opened a new log
I20260619 06:39:46.021262 9385 tablet_bootstrap.cc:492] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01: No bootstrap required, opened a new log
I20260619 06:39:46.021348 9384 ts_tablet_manager.cc:1397] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:39:46.021409 9385 ts_tablet_manager.cc:1397] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:39:46.021369 9383 consensus_queue.cc:260] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:46.021823 9383 ts_tablet_manager.cc:1428] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 06:39:46.022033 9384 raft_consensus.cc:348] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:46.022249 9384 raft_consensus.cc:374] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:46.022315 9384 raft_consensus.cc:729] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd871d765e074093b19093eeb27fdc6e, State: Initialized, Role: FOLLOWER
I20260619 06:39:46.022400 9385 raft_consensus.cc:348] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:46.022631 9385 raft_consensus.cc:374] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:46.021752 9389 tablet_replica.cc:439] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } } }
I20260619 06:39:46.022725 9385 raft_consensus.cc:729] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671c78dfb25742c69bdd432e1430ca01, State: Initialized, Role: FOLLOWER
I20260619 06:39:46.022819 9389 tablet_replica.cc:442] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:46.022790 9384 consensus_queue.cc:260] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:46.022941 9385 consensus_queue.cc:260] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:46.023392 9384 ts_tablet_manager.cc:1428] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:39:46.023448 9385 ts_tablet_manager.cc:1428] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:39:46.023327 9394 tablet_replica.cc:439] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } } }
I20260619 06:39:46.023602 9394 tablet_replica.cc:442] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:46.024230 9390 tablet_replica.cc:439] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } } }
I20260619 06:39:46.024521 9390 tablet_replica.cc:442] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:46.254350 9394 raft_consensus.cc:482] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:39:46.254539 9394 raft_consensus.cc:504] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:46.255282 9394 leader_election.cc:288] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dfccfcc939d459ea922377b61078b13 (127.7.146.194:35967), 671c78dfb25742c69bdd432e1430ca01 (127.7.146.195:43601)
I20260619 06:39:46.255829 9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeaff653b2944008aef99473b84ff5a" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfccfcc939d459ea922377b61078b13" is_pre_election: true
I20260619 06:39:46.255851 9314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeaff653b2944008aef99473b84ff5a" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671c78dfb25742c69bdd432e1430ca01" is_pre_election: true
I20260619 06:39:46.256045 9175 raft_consensus.cc:2454] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd871d765e074093b19093eeb27fdc6e in term 0.
I20260619 06:39:46.256053 9314 raft_consensus.cc:2454] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd871d765e074093b19093eeb27fdc6e in term 0.
I20260619 06:39:46.256433 8967 leader_election.cc:302] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dfccfcc939d459ea922377b61078b13, fd871d765e074093b19093eeb27fdc6e; no voters:
I20260619 06:39:46.256704 9394 raft_consensus.cc:2781] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:39:46.256850 9394 raft_consensus.cc:482] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:39:46.256953 9394 raft_consensus.cc:3037] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:46.259001 9394 raft_consensus.cc:504] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:46.259595 9394 leader_election.cc:288] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 election: Requested vote from peers 9dfccfcc939d459ea922377b61078b13 (127.7.146.194:35967), 671c78dfb25742c69bdd432e1430ca01 (127.7.146.195:43601)
I20260619 06:39:46.260105 9175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeaff653b2944008aef99473b84ff5a" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfccfcc939d459ea922377b61078b13"
I20260619 06:39:46.260149 9314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeaff653b2944008aef99473b84ff5a" candidate_uuid: "fd871d765e074093b19093eeb27fdc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671c78dfb25742c69bdd432e1430ca01"
I20260619 06:39:46.260346 9314 raft_consensus.cc:3037] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:46.260367 9175 raft_consensus.cc:3037] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:46.263185 9175 raft_consensus.cc:2454] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd871d765e074093b19093eeb27fdc6e in term 1.
I20260619 06:39:46.263197 9314 raft_consensus.cc:2454] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd871d765e074093b19093eeb27fdc6e in term 1.
I20260619 06:39:46.263608 8969 leader_election.cc:302] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 671c78dfb25742c69bdd432e1430ca01, 9dfccfcc939d459ea922377b61078b13, fd871d765e074093b19093eeb27fdc6e; no voters:
I20260619 06:39:46.263880 9394 raft_consensus.cc:2781] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:46.264012 9394 raft_consensus.cc:686] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [term 1 LEADER]: Becoming Leader. State: Replica: fd871d765e074093b19093eeb27fdc6e, State: Running, Role: LEADER
I20260619 06:39:46.264246 9394 consensus_queue.cc:237] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } }
I20260619 06:39:46.265730 9391 tablet_replica.cc:439] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: TxnStatusTablet state changed. Reason: New leader fd871d765e074093b19093eeb27fdc6e. Latest consensus state: current_term: 1 leader_uuid: "fd871d765e074093b19093eeb27fdc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } } }
I20260619 06:39:46.265919 9391 tablet_replica.cc:442] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:39:46.266378 9426 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:39:46.266594 9426 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:39:46.266932 8880 catalog_manager.cc:5495] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e reported cstate change: term changed from 0 to 1, leader changed from <none> to fd871d765e074093b19093eeb27fdc6e (127.7.146.193). New cstate: current_term: 1 leader_uuid: "fd871d765e074093b19093eeb27fdc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } health_report { overall_health: UNKNOWN } } }
I20260619 06:39:46.315828 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:46.310371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:41247 (local address 127.0.0.1:43668)
0619 06:39:46.310576 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:46.310594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:46.310709 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:46.311085 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:46.311088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:46.311097 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:39:46.311240 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:46.311247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.312676 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.312681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:46.313687 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:46.313699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.314739 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.314743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:46.314945 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:46.315005 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:39:46.315402 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:39:46.315413 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:39:46.315466 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 06:39:46.315572 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 06:39:46.315874 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:46.310394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.7.146.193:41247)
0619 06:39:46.310591 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:46.310597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:46.310661 (+ 64us) server_negotiation.cc:407] Connection header received
0619 06:39:46.310883 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:46.310890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:46.310973 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:46.311073 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:46.311358 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.312523 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.313847 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.314579 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.314703 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:46.315177 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:46.315310 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:46.315411 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:46.315508 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 06:39:46.315589 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:39:46.319651 9314 raft_consensus.cc:1261] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01 [term 1 FOLLOWER]: Refusing update from remote peer fd871d765e074093b19093eeb27fdc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:46.319746 9175 raft_consensus.cc:1261] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13 [term 1 FOLLOWER]: Refusing update from remote peer fd871d765e074093b19093eeb27fdc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:46.320367 9394 consensus_queue.cc:1035] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:46.320673 9314 raft_consensus.cc:3037] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:46.320799 9391 consensus_queue.cc:1035] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:46.324620 9394 consensus_queue.cc:1035] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:46.330189 9391 consensus_queue.cc:1035] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:46.335251 9389 tablet_replica.cc:439] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13: TxnStatusTablet state changed. Reason: New leader fd871d765e074093b19093eeb27fdc6e. Latest consensus state: current_term: 1 leader_uuid: "fd871d765e074093b19093eeb27fdc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } } }
I20260619 06:39:46.335561 9389 tablet_replica.cc:442] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:46.338061 9391 tablet_replica.cc:439] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd871d765e074093b19093eeb27fdc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } } }
I20260619 06:39:46.338354 9391 tablet_replica.cc:442] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:39:46.343654 9394 tablet_replica.cc:439] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd871d765e074093b19093eeb27fdc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } } }
I20260619 06:39:46.343959 9394 tablet_replica.cc:442] T 6eeaff653b2944008aef99473b84ff5a P fd871d765e074093b19093eeb27fdc6e: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:39:46.344221 9390 tablet_replica.cc:439] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01: TxnStatusTablet state changed. Reason: New leader fd871d765e074093b19093eeb27fdc6e. Latest consensus state: current_term: 1 leader_uuid: "fd871d765e074093b19093eeb27fdc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } } }
I20260619 06:39:46.344559 9390 tablet_replica.cc:442] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:46.355839 9389 tablet_replica.cc:439] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd871d765e074093b19093eeb27fdc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } } }
I20260619 06:39:46.356120 9389 tablet_replica.cc:442] T 6eeaff653b2944008aef99473b84ff5a P 9dfccfcc939d459ea922377b61078b13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:46.360908 9390 tablet_replica.cc:439] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd871d765e074093b19093eeb27fdc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd871d765e074093b19093eeb27fdc6e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41247 } } peers { permanent_uuid: "9dfccfcc939d459ea922377b61078b13" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35967 } } peers { permanent_uuid: "671c78dfb25742c69bdd432e1430ca01" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43601 } } }
I20260619 06:39:46.361249 9390 tablet_replica.cc:442] T 6eeaff653b2944008aef99473b84ff5a P 671c78dfb25742c69bdd432e1430ca01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:46.373370 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:46.366729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:57983 (local address 127.7.146.193:41247)
0619 06:39:46.366919 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:46.366925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:46.367597 (+ 672us) server_negotiation.cc:407] Connection header received
0619 06:39:46.367779 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:46.367783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:46.367856 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:46.367949 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:46.368635 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.369393 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.371448 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.371896 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.372002 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:46.372503 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:46.372661 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:46.372732 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:46.372987 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 06:39:46.373056 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":53}
I20260619 06:39:46.373605 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:46.366730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:41247 (local address 127.7.146.193:57983)
0619 06:39:46.367462 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:46.367485 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:46.367642 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:46.367981 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:46.367986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:46.367999 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:39:46.368415 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:46.368427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.369521 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.369527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:46.371290 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:46.371305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.372058 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.372065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:46.372240 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:46.372321 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:39:46.372829 (+ 508us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:39:46.372848 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:39:46.372936 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:39:46.373100 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":21,"thread_start_us":341,"threads_started":1}
I20260619 06:39:46.436923 8880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:46.437197 8880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:46.440444 8880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef9a5232821c4bcfa54049fa928626e1: Sending DeleteTablet for 3 replicas of tablet bc18edda99a44d9abea3a3192e8c4349
I20260619 06:39:46.441422 9155 tablet_service.cc:1514] Processing DeleteTablet for tablet bc18edda99a44d9abea3a3192e8c4349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:46 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:52898
I20260619 06:39:46.441448 9294 tablet_service.cc:1514] Processing DeleteTablet for tablet bc18edda99a44d9abea3a3192e8c4349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:46 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:39024
I20260619 06:39:46.441888 9015 tablet_service.cc:1514] Processing DeleteTablet for tablet bc18edda99a44d9abea3a3192e8c4349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:46 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:43654
I20260619 06:39:46.442178 9449 tablet_replica.cc:330] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01: stopping tablet replica
I20260619 06:39:46.442436 9449 raft_consensus.cc:2229] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:46.442629 9450 tablet_replica.cc:330] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e: stopping tablet replica
I20260619 06:39:46.442788 9449 raft_consensus.cc:2258] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:46.442896 9450 raft_consensus.cc:2229] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:39:46.443348 9450 raft_consensus.cc:2258] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:46.443948 9449 ts_tablet_manager.cc:1905] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:46.444411 9450 ts_tablet_manager.cc:1905] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:46.451328 9449 ts_tablet_manager.cc:1918] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:39:46.451594 9449 log.cc:1196] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/bc18edda99a44d9abea3a3192e8c4349
I20260619 06:39:46.451885 9450 ts_tablet_manager.cc:1918] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:39:46.452103 9449 ts_tablet_manager.cc:1939] T bc18edda99a44d9abea3a3192e8c4349 P 671c78dfb25742c69bdd432e1430ca01: Deleting consensus metadata
I20260619 06:39:46.452147 9450 log.cc:1196] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/bc18edda99a44d9abea3a3192e8c4349
I20260619 06:39:46.452659 9450 ts_tablet_manager.cc:1939] T bc18edda99a44d9abea3a3192e8c4349 P fd871d765e074093b19093eeb27fdc6e: Deleting consensus metadata
I20260619 06:39:46.453440 8866 catalog_manager.cc:4841] TS 671c78dfb25742c69bdd432e1430ca01 (127.7.146.195:43601): tablet bc18edda99a44d9abea3a3192e8c4349 (table test-table [id=49d42efdddcc4dea91be80cbe4b3450a]) successfully deleted
I20260619 06:39:46.453863 8867 catalog_manager.cc:4841] TS fd871d765e074093b19093eeb27fdc6e (127.7.146.193:41247): tablet bc18edda99a44d9abea3a3192e8c4349 (table test-table [id=49d42efdddcc4dea91be80cbe4b3450a]) successfully deleted
I20260619 06:39:46.456986 9448 tablet_replica.cc:330] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13: stopping tablet replica
I20260619 06:39:46.457199 9448 raft_consensus.cc:2229] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:46.457466 9448 raft_consensus.cc:2258] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:46.458606 9448 ts_tablet_manager.cc:1905] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:46.463860 9448 ts_tablet_manager.cc:1918] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:39:46.464040 9448 log.cc:1196] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/bc18edda99a44d9abea3a3192e8c4349
I20260619 06:39:46.464437 9448 ts_tablet_manager.cc:1939] T bc18edda99a44d9abea3a3192e8c4349 P 9dfccfcc939d459ea922377b61078b13: Deleting consensus metadata
I20260619 06:39:46.465332 8864 catalog_manager.cc:4841] TS 9dfccfcc939d459ea922377b61078b13 (127.7.146.194:35967): tablet bc18edda99a44d9abea3a3192e8c4349 (table test-table [id=49d42efdddcc4dea91be80cbe4b3450a]) successfully deleted
Jun 19 06:39:46 dist-test-slave-qfk4 krb5kdc[8835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851185, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:39:46.480105 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:46.458751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.7.146.193:41247)
0619 06:39:46.458936 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:46.458943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:46.463760 (+ 4817us) server_negotiation.cc:407] Connection header received
0619 06:39:46.463955 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:46.463961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:46.464048 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:46.464178 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:46.466214 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.467094 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.468945 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.469400 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.469501 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:46.474253 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:46.474269 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:46.474284 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:46.474338 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:46.477372 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:46.478200 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:46.478207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:46.478209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:46.478342 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:46.478715 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:46.478718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:46.478720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:46.479080 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:46.479310 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:46.479602 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:39:46.479784 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 06:39:46.480871 9081 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43672
I20260619 06:39:46.495431 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:46.483528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.7.146.254:33073)
0619 06:39:46.483716 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:46.483722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:46.483961 (+ 239us) server_negotiation.cc:407] Connection header received
0619 06:39:46.484132 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:46.484136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:46.484199 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:46.484280 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:46.485598 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.486363 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.487529 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.487950 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.488057 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:46.490320 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:46.490325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:46.490329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:46.490364 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:46.492981 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:46.493604 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:46.493609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:46.493613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:46.493729 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:46.494068 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:46.494073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:46.494077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:46.494348 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:46.494568 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:46.494925 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 06:39:46.495134 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 06:39:46.496033 8880 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45196
I20260619 06:39:46.498027 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 8951
I20260619 06:39:46.511297 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:46.499111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.7.146.193:41247)
0619 06:39:46.499291 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:46.499297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:46.499314 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:39:46.499370 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:46.499375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:46.499450 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:46.499554 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:46.500558 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.501750 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.502887 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.503490 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.503607 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:46.505684 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:46.505690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:46.505702 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:46.505740 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:46.508835 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:46.509452 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:46.509458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:46.509462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:46.509639 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:46.509932 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:46.509937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:46.509941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:46.510234 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:46.510501 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:46.510690 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 06:39:46.510911 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 06:39:46.511904 9081 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43674
W20260619 06:39:46.512401 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 8951: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:46.512598 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 8951
I20260619 06:39:46.530395 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 9090
Jun 19 06:39:46 dist-test-slave-qfk4 krb5kdc[8835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851185, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:39:46.545174 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:46.531188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.7.146.194:35967)
0619 06:39:46.531608 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:46.531613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:46.531629 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:39:46.531683 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:46.531687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:46.531790 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:46.531882 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:46.533088 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.533811 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.535045 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.535465 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.535555 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:46.539629 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:46.539645 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:46.539662 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:46.539697 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:46.542574 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:46.543301 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:46.543308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:46.543310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:46.543452 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:46.543811 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:46.543815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:46.543817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:46.544168 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:46.544383 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:46.544654 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 06:39:46.544836 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":199,"threads_started":1}
W20260619 06:39:46.545828 9220 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52914
W20260619 06:39:46.546423 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 9090: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:46.546598 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 9090
I20260619 06:39:46.559391 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 9229
Jun 19 06:39:46 dist-test-slave-qfk4 krb5kdc[8835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851185, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:39:46.573746 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:46.560066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.7.146.195:43601)
0619 06:39:46.560547 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:46.560551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:46.560599 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:39:46.560656 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:46.560659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:46.560718 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:46.560810 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:46.561956 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.562679 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.563910 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:46.564317 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:46.564409 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:46.568615 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:46.568632 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:46.568649 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:46.568691 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:46.571296 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:46.571913 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:46.571920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:46.571922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:46.572044 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:46.572333 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:46.572336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:46.572338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:46.572746 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:46.572958 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:46.573298 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 06:39:46.573468 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":259,"threads_started":1}
W20260619 06:39:46.574393 9359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39036
W20260619 06:39:46.574986 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 9229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:46.575120 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 9229
I20260619 06:39:46.587955 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 8848
I20260619 06:39:46.589143 8920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:46.702049 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 8848
2026-06-19T06:39:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4678 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:46 dist-test-slave-qfk4 krb5kdc[9461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:46 dist-test-slave-qfk4 krb5kdc[9461](info): set up 2 sockets
Jun 19 06:39:46 dist-test-slave-qfk4 krb5kdc[9461](info): commencing operation
krb5kdc: starting...
W20260619 06:39:47.696666 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:39:47 dist-test-slave-qfk4 krb5kdc[9461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:39:47.866259 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:33771
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:33771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:48.172434 9474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:48.172811 9474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:48.172959 9474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:48.180303 9474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:48.180442 9474 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:39:48.180513 9474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:48.180619 9474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:48.180694 9474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:48.180778 9474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:48.190779 9474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:33771
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:33771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:48.192020 9474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:48.193334 9474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:48.204202 9479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:48.204211 9480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:48.204704 9482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:48.205195 9474 server_base.cc:1034] running on GCE node
I20260619 06:39:48.205945 9474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:48.207254 9474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:48.208490 9474 hybrid_clock.cc:648] HybridClock initialized: now 1781851188208421 us; error 27 us; skew 500 ppm
I20260619 06:39:48.208909 9474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:48 dist-test-slave-qfk4 krb5kdc[9461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851188, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:48.215129 9474 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:39:48.216872 9474 webserver.cc:460] Webserver started at http://127.7.146.254:44975/ using document root <none> and password file <none>
I20260619 06:39:48.217574 9474 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:48.217669 9474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:48.217981 9474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:48.221668 9474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "5c68d2f1684748f69c02da8ed0148b84"
format_stamp: "Formatted at 2026-06-19 06:39:48 on dist-test-slave-qfk4"
server_key: "2e56840391b5fe0e065313ea3bffda8a"
server_key_iv: "7c4ac818ac80a3cee67493addfd128a8"
server_key_version: "encryptionkey@0"
I20260619 06:39:48.222543 9474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "5c68d2f1684748f69c02da8ed0148b84"
format_stamp: "Formatted at 2026-06-19 06:39:48 on dist-test-slave-qfk4"
server_key: "2e56840391b5fe0e065313ea3bffda8a"
server_key_iv: "7c4ac818ac80a3cee67493addfd128a8"
server_key_version: "encryptionkey@0"
I20260619 06:39:48.229272 9474 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 06:39:48.232965 9489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:48.234539 9474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:48.234802 9474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "5c68d2f1684748f69c02da8ed0148b84"
format_stamp: "Formatted at 2026-06-19 06:39:48 on dist-test-slave-qfk4"
server_key: "2e56840391b5fe0e065313ea3bffda8a"
server_key_iv: "7c4ac818ac80a3cee67493addfd128a8"
server_key_version: "encryptionkey@0"
I20260619 06:39:48.235105 9474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:48.282460 9474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:48.286337 9474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:48.286545 9474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:48.306023 9474 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:33771
I20260619 06:39:48.306035 9550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:33771 every 8 connection(s)
I20260619 06:39:48.307307 9474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:48.310909 9551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:48.312222 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 9474
I20260619 06:39:48.312398 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:39:48.313098 7755 external_mini_cluster.cc:1428] Setting key 047cae29bb9fd4242c7939c011d5f0a0
I20260619 06:39:48.322192 9551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84: Bootstrap starting.
I20260619 06:39:48.325672 9551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:48.326896 9551 log.cc:824] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:39:48 dist-test-slave-qfk4 krb5kdc[9461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:48.329994 9551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84: No bootstrap required, opened a new log
I20260619 06:39:48.336403 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:48.315232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.7.146.254:33771)
0619 06:39:48.316032 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:48.316045 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:48.316097 (+ 52us) server_negotiation.cc:407] Connection header received
0619 06:39:48.317863 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:48.317887 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:48.318272 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:48.318619 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:48.320270 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:48.321650 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:48.323394 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:48.324190 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:48.324340 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:48.329100 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:48.329108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:48.329121 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:48.329156 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:48.332179 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:48.333051 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:48.333058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:48.333064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:48.333260 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:48.333669 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:48.333673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:48.333676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:48.334335 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:48.334615 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:48.334997 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 06:39:48.335363 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":374,"threads_started":1}
I20260619 06:39:48.338573 9551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68d2f1684748f69c02da8ed0148b84" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33771 } }
I20260619 06:39:48.338914 9551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:48.339018 9551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c68d2f1684748f69c02da8ed0148b84, State: Initialized, Role: FOLLOWER
I20260619 06:39:48.339574 9551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68d2f1684748f69c02da8ed0148b84" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33771 } }
I20260619 06:39:48.339821 9551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:48.339941 9551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:48.340090 9551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:48.342268 9551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68d2f1684748f69c02da8ed0148b84" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33771 } }
I20260619 06:39:48.342705 9551 leader_election.cc:302] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c68d2f1684748f69c02da8ed0148b84; no voters:
I20260619 06:39:48.343289 9551 leader_election.cc:288] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:48.343426 9556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:48.344206 9556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [term 1 LEADER]: Becoming Leader. State: Replica: 5c68d2f1684748f69c02da8ed0148b84, State: Running, Role: LEADER
I20260619 06:39:48.344743 9556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68d2f1684748f69c02da8ed0148b84" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33771 } }
I20260619 06:39:48.345415 9551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:48.347363 9558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c68d2f1684748f69c02da8ed0148b84. Latest consensus state: current_term: 1 leader_uuid: "5c68d2f1684748f69c02da8ed0148b84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68d2f1684748f69c02da8ed0148b84" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33771 } } }
I20260619 06:39:48.347764 9558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:48.349291 9565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:48.349165 9557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c68d2f1684748f69c02da8ed0148b84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c68d2f1684748f69c02da8ed0148b84" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33771 } } }
I20260619 06:39:48.349538 9557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:48.354140 9565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:48.362670 9565 catalog_manager.cc:1347] Generated new cluster ID: 7a2c0c098f504cb39da28a10a58a6dab
I20260619 06:39:48.362787 9565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:48.384123 9565 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:48.385452 9565 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:48.395556 9565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84: Generated new TSK 0
I20260619 06:39:48.396374 9565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:39:48.449270 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33771
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:48.776990 9577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:48.777253 9577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:48.777376 9577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:48.784502 9577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:48.784646 9577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:48.784781 9577 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:39:48.784840 9577 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:39:48.784914 9577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:39:48.794636 9577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.146.254:33771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:48.796001 9577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:48.797323 9577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:48.809653 9582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:48.809585 9585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:48.809620 9583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:48.810539 9577 server_base.cc:1034] running on GCE node
I20260619 06:39:48.811167 9577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:48.812079 9577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:48.813261 9577 hybrid_clock.cc:648] HybridClock initialized: now 1781851188813225 us; error 40 us; skew 500 ppm
I20260619 06:39:48.813547 9577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:48 dist-test-slave-qfk4 krb5kdc[9461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851188, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:48.816459 9577 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:39:48.817992 9577 webserver.cc:460] Webserver started at http://127.7.146.193:43857/ using document root <none> and password file <none>
I20260619 06:39:48.818614 9577 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:48.818723 9577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:48.819008 9577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:48.821780 9577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "de2dafd092184cfda75e35989d449921"
format_stamp: "Formatted at 2026-06-19 06:39:48 on dist-test-slave-qfk4"
server_key: "64c39a93ae665989cd96e19f0767188a"
server_key_iv: "d5a0348d9c3ddf70f15ab7fdf1fbf497"
server_key_version: "encryptionkey@0"
I20260619 06:39:48.822474 9577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "de2dafd092184cfda75e35989d449921"
format_stamp: "Formatted at 2026-06-19 06:39:48 on dist-test-slave-qfk4"
server_key: "64c39a93ae665989cd96e19f0767188a"
server_key_iv: "d5a0348d9c3ddf70f15ab7fdf1fbf497"
server_key_version: "encryptionkey@0"
I20260619 06:39:48.827422 9577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:39:48.830848 9592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:48.832121 9577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:39:48.832265 9577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "de2dafd092184cfda75e35989d449921"
format_stamp: "Formatted at 2026-06-19 06:39:48 on dist-test-slave-qfk4"
server_key: "64c39a93ae665989cd96e19f0767188a"
server_key_iv: "d5a0348d9c3ddf70f15ab7fdf1fbf497"
server_key_version: "encryptionkey@0"
I20260619 06:39:48.832446 9577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:48.845748 9577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:48.850071 9577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:48.850337 9577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:48.852906 9577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:48.853011 9577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:48.853125 9577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:48.853195 9577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:39:48 dist-test-slave-qfk4 krb5kdc[9461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851188, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:48.879115 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:48.853471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:60305 (local address 127.7.146.254:33771)
0619 06:39:48.860916 (+ 7445us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:48.860921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:48.860939 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:39:48.861012 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:48.861017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:48.861101 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:48.861223 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:48.863842 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:48.864759 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:48.867522 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:48.867918 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:48.868016 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:48.872426 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:48.872433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:48.872438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:48.872480 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:48.875285 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:48.876224 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:48.876230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:48.876234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:48.876397 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:48.877047 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:48.877053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:48.877057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:48.877441 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:48.877584 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:48.878533 (+ 949us) server_negotiation.cc:299] Negotiation successful
0619 06:39:48.878752 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7331,"thread_start_us":7245,"threads_started":1}
I20260619 06:39:48.880167 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:48.854217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33771 (local address 127.7.146.193:60305)
0619 06:39:48.855177 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:48.855224 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:48.857820 (+ 2596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:48.861575 (+ 3755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:48.861588 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:48.862576 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:48.863569 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:48.863601 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:48.864938 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:48.864948 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:48.867374 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:48.867391 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:48.868143 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:48.868149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:48.868450 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:48.869191 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:48.869221 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:48.872204 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:48.875418 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:48.875428 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:48.875448 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:48.876084 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:48.876551 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:48.876556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:48.876582 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:48.876885 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:48.877603 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:48.877612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:48.878082 (+ 470us) client_negotiation.cc:769] Sending connection context
0619 06:39:48.878439 (+ 357us) client_negotiation.cc:240] Negotiation successful
0619 06:39:48.878996 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":373,"threads_started":1}
I20260619 06:39:48.900487 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:48.886662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:42871 (local address 127.7.146.254:33771)
0619 06:39:48.886820 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:48.886826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:48.887705 (+ 879us) server_negotiation.cc:407] Connection header received
0619 06:39:48.887954 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:48.887960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:48.888043 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:48.888142 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:48.889604 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:48.890538 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:48.892118 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:48.892611 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:48.892739 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:48.894961 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:48.894968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:48.894974 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:48.895014 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:48.898132 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:48.898746 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:48.898751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:48.898755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:48.898873 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:48.899171 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:48.899176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:48.899180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:48.899449 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:48.899579 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:48.899939 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 06:39:48.900140 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:39:48.900609 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:48.886644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33771 (local address 127.7.146.193:42871)
0619 06:39:48.887604 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:48.887622 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:48.887769 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:48.888341 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:48.888346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:48.889030 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:48.889422 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:48.889435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:48.890708 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:48.890717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:48.891988 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:48.892001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:48.892792 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:48.892801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:48.892998 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:48.893711 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:48.893743 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:48.894780 (+ 1037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:48.898254 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:48.898259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:48.898262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:48.898642 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:48.898968 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:48.898971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:48.898974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:48.899080 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:48.899554 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:48.899559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:48.899728 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 06:39:48.899840 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 06:39:48.900132 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":512,"threads_started":1}
I20260619 06:39:48.904781 9577 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:45073
I20260619 06:39:48.904821 9712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:45073 every 8 connection(s)
I20260619 06:39:48.905998 9577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:39:48.907902 9713 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33771
I20260619 06:39:48.908105 9713 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:48.908885 9713 heartbeater.cc:511] Master 127.7.146.254:33771 requested a full tablet report, sending...
I20260619 06:39:48.908888 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 9577
I20260619 06:39:48.909073 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:39:48.909828 7755 external_mini_cluster.cc:1428] Setting key 4ee9b0b9844c73a3e7bccbb52d4d32a0
I20260619 06:39:48.911185 9506 ts_manager.cc:194] Registered new tserver with Master: de2dafd092184cfda75e35989d449921 (127.7.146.193:45073)
I20260619 06:39:48.913074 9506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:60305
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:39:48.949867 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33771
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:49.255414 9717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:49.255692 9717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:49.255820 9717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:49.262934 9717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:49.263046 9717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:49.263150 9717 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:39:49.263209 9717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:39:49.263296 9717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:39:49.272931 9717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.146.254:33771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:49.274176 9717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:49.275447 9717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:49.288079 9723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:49.288475 9722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:49.288141 9725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:49.289027 9717 server_base.cc:1034] running on GCE node
I20260619 06:39:49.289631 9717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:49.290673 9717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:49.291867 9717 hybrid_clock.cc:648] HybridClock initialized: now 1781851189291818 us; error 46 us; skew 500 ppm
I20260619 06:39:49.292157 9717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:49 dist-test-slave-qfk4 krb5kdc[9461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851189, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:49.295055 9717 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:39:49.296442 9717 webserver.cc:460] Webserver started at http://127.7.146.194:46433/ using document root <none> and password file <none>
I20260619 06:39:49.297111 9717 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:49.297209 9717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:49.297462 9717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:49.300208 9717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "5ad615111bc34050a01200f47afbfae1"
format_stamp: "Formatted at 2026-06-19 06:39:49 on dist-test-slave-qfk4"
server_key: "74d7bbe6873422348b06639a5aa67a89"
server_key_iv: "04bdb3c5856c87778bc4edb93f27127b"
server_key_version: "encryptionkey@0"
I20260619 06:39:49.300952 9717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "5ad615111bc34050a01200f47afbfae1"
format_stamp: "Formatted at 2026-06-19 06:39:49 on dist-test-slave-qfk4"
server_key: "74d7bbe6873422348b06639a5aa67a89"
server_key_iv: "04bdb3c5856c87778bc4edb93f27127b"
server_key_version: "encryptionkey@0"
I20260619 06:39:49.306064 9717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:39:49.310228 9732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:49.311686 9717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:39:49.311844 9717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "5ad615111bc34050a01200f47afbfae1"
format_stamp: "Formatted at 2026-06-19 06:39:49 on dist-test-slave-qfk4"
server_key: "74d7bbe6873422348b06639a5aa67a89"
server_key_iv: "04bdb3c5856c87778bc4edb93f27127b"
server_key_version: "encryptionkey@0"
I20260619 06:39:49.312042 9717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:49.336475 9717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:49.340605 9717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:49.340848 9717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:49.343437 9717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:49.343557 9717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:49.343665 9717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:49.343750 9717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:39:49 dist-test-slave-qfk4 krb5kdc[9461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851189, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:49.365607 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.344058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:59719 (local address 127.7.146.254:33771)
0619 06:39:49.344257 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:49.344263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:49.345877 (+ 1614us) server_negotiation.cc:407] Connection header received
0619 06:39:49.347942 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:49.347948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:49.348061 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:49.348147 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:49.350322 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.351183 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.354152 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.354747 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.354890 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.359004 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:49.359012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:49.359017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:49.359063 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:49.362443 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:49.363162 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:49.363168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:49.363172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:49.363331 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:49.363701 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:49.363706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:49.363709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:49.364066 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:49.364226 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:49.365043 (+ 817us) server_negotiation.cc:299] Negotiation successful
0619 06:39:49.365278 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:39:49.366986 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.344836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33771 (local address 127.7.146.194:59719)
0619 06:39:49.345661 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:49.345763 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:49.347649 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:49.348427 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:49.348439 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:49.349250 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:49.350060 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.350100 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.351435 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.351443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.353971 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.353991 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.355028 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.355035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.355345 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.356104 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:49.356128 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:49.358735 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:49.362606 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:49.362614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:49.362628 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:49.363046 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:49.363427 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:49.363431 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:49.363433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:49.363611 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:49.364205 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:49.364214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:49.364553 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 06:39:49.364946 (+ 393us) client_negotiation.cc:240] Negotiation successful
0619 06:39:49.365476 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":318,"threads_started":1}
I20260619 06:39:49.389760 9717 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:39145
I20260619 06:39:49.389798 9851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:39145 every 8 connection(s)
I20260619 06:39:49.391170 9717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:39:49.393199 9852 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33771
I20260619 06:39:49.393429 9852 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:49.394181 9852 heartbeater.cc:511] Master 127.7.146.254:33771 requested a full tablet report, sending...
I20260619 06:39:49.395609 9506 ts_manager.cc:194] Registered new tserver with Master: 5ad615111bc34050a01200f47afbfae1 (127.7.146.194:39145)
I20260619 06:39:49.395704 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.378584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33771 (local address 127.7.146.194:54647)
0619 06:39:49.381364 (+ 2780us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:49.381388 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:49.381665 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:49.382058 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:49.382064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:49.382705 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:49.383313 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.383331 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.384972 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.384978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.386896 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.386917 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.387919 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.387926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.388078 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.388737 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:49.388763 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:49.389894 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:49.393282 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:49.393294 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:49.393298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:49.393829 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:49.394182 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:49.394186 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:49.394190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:49.394339 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:49.394848 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:49.394853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:49.395017 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 06:39:49.395153 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 06:39:49.395410 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2306,"thread_start_us":2137,"threads_started":1}
I20260619 06:39:49.396343 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 9717
I20260619 06:39:49.396505 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:39:49.396713 9506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:59719
I20260619 06:39:49.396898 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.379015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:54647 (local address 127.7.146.254:33771)
0619 06:39:49.379260 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:49.379267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:49.381487 (+ 2220us) server_negotiation.cc:407] Connection header received
0619 06:39:49.381834 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:49.381838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:49.381913 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:49.381989 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:49.383549 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.384801 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.387055 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.387685 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.387835 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.390123 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:49.390128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:49.390132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:49.390171 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:49.393161 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:49.393936 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:49.393940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:49.393942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:49.394070 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:49.394430 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:49.394433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:49.394435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:49.394734 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:49.394842 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:49.395302 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 06:39:49.395512 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:39:49.397073 7755 external_mini_cluster.cc:1428] Setting key 5efd91ccad1e081ea12c49b0708c50a3
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:39:49.433777 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33771
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:49.738114 9856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:49.738406 9856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:49.738547 9856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:49.745882 9856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:49.746006 9856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:49.746126 9856 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:39:49.746191 9856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:39:49.746289 9856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:39:49.756541 9856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.146.254:33771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:49.757897 9856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:49.759163 9856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:49.771957 9862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:49.772064 9864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:49.772004 9861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:49.772869 9856 server_base.cc:1034] running on GCE node
I20260619 06:39:49.773459 9856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:49.774361 9856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:49.775590 9856 hybrid_clock.cc:648] HybridClock initialized: now 1781851189775538 us; error 43 us; skew 500 ppm
I20260619 06:39:49.775990 9856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:49 dist-test-slave-qfk4 krb5kdc[9461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851189, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:49.783370 9856 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:39:49.785037 9856 webserver.cc:460] Webserver started at http://127.7.146.195:42551/ using document root <none> and password file <none>
I20260619 06:39:49.785724 9856 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:49.785846 9856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:49.786212 9856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:49.789439 9856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "24e737c9ecba4e98b1b74bc79d42a49f"
format_stamp: "Formatted at 2026-06-19 06:39:49 on dist-test-slave-qfk4"
server_key: "8422d7a7b1d78e1a738782e7e3a184cc"
server_key_iv: "d9f49f22113f02c76d4ad3587690a6b8"
server_key_version: "encryptionkey@0"
I20260619 06:39:49.790148 9856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "24e737c9ecba4e98b1b74bc79d42a49f"
format_stamp: "Formatted at 2026-06-19 06:39:49 on dist-test-slave-qfk4"
server_key: "8422d7a7b1d78e1a738782e7e3a184cc"
server_key_iv: "d9f49f22113f02c76d4ad3587690a6b8"
server_key_version: "encryptionkey@0"
I20260619 06:39:49.795208 9856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:39:49.798878 9871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:49.800429 9856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:39:49.800601 9856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "24e737c9ecba4e98b1b74bc79d42a49f"
format_stamp: "Formatted at 2026-06-19 06:39:49 on dist-test-slave-qfk4"
server_key: "8422d7a7b1d78e1a738782e7e3a184cc"
server_key_iv: "d9f49f22113f02c76d4ad3587690a6b8"
server_key_version: "encryptionkey@0"
I20260619 06:39:49.800809 9856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:49.815235 9856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:49.819528 9856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:49.819806 9856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:49.822607 9856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:49.822751 9856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:49.822871 9856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:49.822939 9856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:39:49 dist-test-slave-qfk4 krb5kdc[9461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851189, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:49.845537 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.822958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:49409 (local address 127.7.146.254:33771)
0619 06:39:49.823184 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:49.823192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:49.824615 (+ 1423us) server_negotiation.cc:407] Connection header received
0619 06:39:49.826627 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:49.826634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:49.826737 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:49.826873 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:49.829152 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.830097 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.832686 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.833478 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.833639 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.837904 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:49.837913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:49.837920 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:49.837974 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:49.841792 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:49.842591 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:49.842598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:49.842602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:49.842793 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:49.843218 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:49.843224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:49.843228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:49.843672 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:49.843842 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:49.844779 (+ 937us) server_negotiation.cc:299] Negotiation successful
0619 06:39:49.845051 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:39:49.847121 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.823573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33771 (local address 127.7.146.195:49409)
0619 06:39:49.824405 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:49.824458 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:49.826328 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:49.827189 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:49.827204 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:49.827961 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:49.828955 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.828987 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.830248 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.830252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.832455 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.832472 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.833715 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.833720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.833927 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.834633 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:49.834656 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:49.837620 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:49.841965 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:49.841973 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:49.841987 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:49.842462 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:49.842905 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:49.842908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:49.842910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:49.843105 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:49.843904 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:49.843911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:49.844291 (+ 380us) client_negotiation.cc:769] Sending connection context
0619 06:39:49.844546 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 06:39:49.845287 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":345,"threads_started":1}
I20260619 06:39:49.866417 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.852522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:41641 (local address 127.7.146.254:33771)
0619 06:39:49.852732 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:49.852739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:49.853248 (+ 509us) server_negotiation.cc:407] Connection header received
0619 06:39:49.853403 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:49.853410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:49.853499 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:49.853621 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:49.854499 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.855757 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.857704 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.858239 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.858362 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.860742 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:49.860747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:49.860750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:49.860781 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:49.863622 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:49.864443 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:49.864448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:49.864452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:49.864624 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:49.865114 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:49.865118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:49.865120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:49.865386 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:49.865492 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:49.865987 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 06:39:49.866149 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:39:49.866876 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.852532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33771 (local address 127.7.146.195:41641)
0619 06:39:49.853161 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:49.853178 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:49.853297 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:49.853640 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:49.853643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:49.854069 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:49.854365 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.854374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.855971 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.855976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.857568 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.857583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.858434 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.858440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.858627 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.859418 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:49.859438 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:49.860514 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:49.863766 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:49.863773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:49.863776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:49.864323 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:49.864752 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:49.864757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:49.864768 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:49.864969 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:49.865494 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:49.865500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:49.865736 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 06:39:49.865907 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 06:39:49.866303 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":318,"threads_started":1}
I20260619 06:39:49.880854 9856 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:36799
I20260619 06:39:49.880919 9990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:36799 every 8 connection(s)
I20260619 06:39:49.882263 9856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:39:49.884589 9991 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33771
I20260619 06:39:49.884838 9991 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:49.885494 9991 heartbeater.cc:511] Master 127.7.146.254:33771 requested a full tablet report, sending...
I20260619 06:39:49.886860 9506 ts_manager.cc:194] Registered new tserver with Master: 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799)
I20260619 06:39:49.887662 9506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:49409
I20260619 06:39:49.890076 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 9856
I20260619 06:39:49.890218 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:39:49.890623 7755 external_mini_cluster.cc:1428] Setting key ae08fd8d9bfda43059ada8cdc98baee6
I20260619 06:39:49.893458 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:39:49 dist-test-slave-qfk4 krb5kdc[9461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260619 06:39:49.916586 9713 heartbeater.cc:503] Master 127.7.146.254:33771 was elected leader, sending a full tablet report...
Jun 19 06:39:49 dist-test-slave-qfk4 krb5kdc[9461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:49.937733 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.924409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.7.146.254:33771)
0619 06:39:49.924611 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:49.924617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:49.924929 (+ 312us) server_negotiation.cc:407] Connection header received
0619 06:39:49.925102 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:49.925106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:49.925170 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:49.925245 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:49.926537 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.927273 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.928449 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.928929 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.929021 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.932743 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:49.932748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:49.932751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:49.932784 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:49.935209 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:49.935935 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:49.935939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:49.935941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:49.936107 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:49.936471 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:49.936474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:49.936476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:49.936781 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:49.936965 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:49.937220 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 06:39:49.937393 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:39:49.941845 9506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:39:49.949288 9506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:39:49.986861 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.961776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:45073 (local address 127.0.0.1:33224)
0619 06:39:49.978850 (+ 17074us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:49.978871 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:49.979024 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:49.980014 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:49.980023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:49.980052 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:49.980497 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.980510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.982291 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.982296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.984331 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.984344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.985949 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.985956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.986143 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.986200 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 06:39:49.986318 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 06:39:49.986492 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16923,"thread_start_us":17096,"threads_started":1}
I20260619 06:39:49.987623 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.963213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:39145 (local address 127.0.0.1:56984)
0619 06:39:49.979160 (+ 15947us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:49.979179 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:49.979393 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:49.980274 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:49.980277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:49.980300 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:49.980761 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.980773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.982426 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.982431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.984911 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.984926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.986827 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.986833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.986994 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.987029 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:39:49.987097 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:39:49.987235 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15822,"mutex_wait_us":2,"thread_start_us":15328,"threads_started":1}
I20260619 06:39:49.987896 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.962073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.7.146.193:45073)
0619 06:39:49.978191 (+ 16118us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:49.978201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:49.979157 (+ 956us) server_negotiation.cc:407] Connection header received
0619 06:39:49.979267 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:49.979275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:49.979547 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:49.979769 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:49.980722 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.982096 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.984485 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.985755 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.987100 (+ 1345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.987295 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:49.987365 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 06:39:49.987527 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15939,"thread_start_us":16130,"threads_started":1}
I20260619 06:39:49.988195 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.961653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:36799 (local address 127.0.0.1:44558)
0619 06:39:49.976912 (+ 15259us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:49.976978 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:49.977232 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:49.981019 (+ 3787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:49.981024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:49.981046 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:49.981368 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.981379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.983232 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.983237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.985393 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:49.985404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.987206 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.987213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:49.987385 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.987416 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:39:49.987500 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 06:39:49.987593 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14855,"mutex_wait_us":4,"thread_start_us":15814,"threads_started":1}
I20260619 06:39:49.988353 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.963702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.7.146.194:39145)
0619 06:39:49.978663 (+ 14961us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:49.978674 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:49.979281 (+ 607us) server_negotiation.cc:407] Connection header received
0619 06:39:49.979572 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:49.979581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:49.979862 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:49.980022 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:49.980937 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.982273 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.985072 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.986360 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.986483 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.986693 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:49.987907 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0619 06:39:49.988075 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14778,"thread_start_us":14632,"threads_started":1}
I20260619 06:39:49.989174 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:49.961944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.7.146.195:36799)
0619 06:39:49.980472 (+ 18528us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:49.980481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:49.980512 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:39:49.980642 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:49.980650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:49.980833 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:49.981026 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:49.981560 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.983080 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.985530 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:49.986737 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:49.986869 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:49.987074 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:49.987701 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 06:39:49.987866 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18322,"thread_start_us":18152,"threads_started":1}
I20260619 06:39:49.992136 9921 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef23fbcc08949ef9fffa466400369db (DEFAULT_TABLE table=test-table [id=38ed70a2d136424e865522d046342b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:49.991983 9643 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef23fbcc08949ef9fffa466400369db (DEFAULT_TABLE table=test-table [id=38ed70a2d136424e865522d046342b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:49.991983 9782 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef23fbcc08949ef9fffa466400369db (DEFAULT_TABLE table=test-table [id=38ed70a2d136424e865522d046342b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:49.993283 9921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef23fbcc08949ef9fffa466400369db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:49.993360 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef23fbcc08949ef9fffa466400369db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:49.993392 9782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef23fbcc08949ef9fffa466400369db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:50.004628 10010 tablet_bootstrap.cc:492] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: Bootstrap starting.
I20260619 06:39:50.007818 10012 tablet_bootstrap.cc:492] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: Bootstrap starting.
I20260619 06:39:50.008481 10010 tablet_bootstrap.cc:654] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:50.009204 10011 tablet_bootstrap.cc:492] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: Bootstrap starting.
I20260619 06:39:50.010015 10010 log.cc:824] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:50.012537 10011 tablet_bootstrap.cc:654] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:50.012820 10012 tablet_bootstrap.cc:654] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:50.013504 10010 tablet_bootstrap.cc:492] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: No bootstrap required, opened a new log
I20260619 06:39:50.013805 10010 ts_tablet_manager.cc:1397] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:39:50.013885 10012 log.cc:824] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:50.013860 10011 log.cc:824] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:50.017047 10011 tablet_bootstrap.cc:492] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: No bootstrap required, opened a new log
I20260619 06:39:50.017096 10012 tablet_bootstrap.cc:492] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: No bootstrap required, opened a new log
I20260619 06:39:50.017379 10012 ts_tablet_manager.cc:1397] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260619 06:39:50.017397 10011 ts_tablet_manager.cc:1397] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20260619 06:39:50.024029 10010 raft_consensus.cc:348] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } }
I20260619 06:39:50.024508 10010 raft_consensus.cc:374] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:50.024657 10010 raft_consensus.cc:729] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e737c9ecba4e98b1b74bc79d42a49f, State: Initialized, Role: FOLLOWER
I20260619 06:39:50.024770 10011 raft_consensus.cc:348] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } }
I20260619 06:39:50.025144 10011 raft_consensus.cc:374] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:50.025238 10011 raft_consensus.cc:729] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad615111bc34050a01200f47afbfae1, State: Initialized, Role: FOLLOWER
I20260619 06:39:50.025308 10010 consensus_queue.cc:260] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } }
I20260619 06:39:50.025785 10011 consensus_queue.cc:260] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } }
I20260619 06:39:50.026556 9991 heartbeater.cc:503] Master 127.7.146.254:33771 was elected leader, sending a full tablet report...
I20260619 06:39:50.027079 10010 ts_tablet_manager.cc:1428] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20260619 06:39:50.027323 10012 raft_consensus.cc:348] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } }
I20260619 06:39:50.027701 10012 raft_consensus.cc:374] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:50.027786 10012 raft_consensus.cc:729] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2dafd092184cfda75e35989d449921, State: Initialized, Role: FOLLOWER
I20260619 06:39:50.027952 9852 heartbeater.cc:503] Master 127.7.146.254:33771 was elected leader, sending a full tablet report...
I20260619 06:39:50.028439 10012 consensus_queue.cc:260] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } }
I20260619 06:39:50.028618 10011 ts_tablet_manager.cc:1428] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: Time spent starting tablet: real 0.011s user 0.001s sys 0.009s
I20260619 06:39:50.030418 10012 ts_tablet_manager.cc:1428] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260619 06:39:50.133951 9992 tablet.cc:2369] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:39:50.143589 9853 tablet.cc:2369] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:39:50.158241 9714 tablet.cc:2369] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:39:50.168363 10017 raft_consensus.cc:482] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:39:50.168803 10017 raft_consensus.cc:504] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } }
I20260619 06:39:50.170897 10017 leader_election.cc:288] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799), de2dafd092184cfda75e35989d449921 (127.7.146.193:45073)
I20260619 06:39:50.180194 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.171992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:39805 (local address 127.7.146.193:45073)
0619 06:39:50.172207 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:50.172213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:50.173343 (+ 1130us) server_negotiation.cc:407] Connection header received
0619 06:39:50.173540 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:50.173545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:50.173625 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:50.173718 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:50.174268 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.174999 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.177818 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.179010 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.179160 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.179236 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:50.179833 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 06:39:50.179961 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:39:50.180194 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.171895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:48489 (local address 127.7.146.195:36799)
0619 06:39:50.172103 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:50.172110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:50.172853 (+ 743us) server_negotiation.cc:407] Connection header received
0619 06:39:50.173163 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:50.173170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:50.173250 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:50.173355 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:50.174076 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.175056 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.177756 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.178979 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.179168 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.179236 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:50.179798 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 06:39:50.179933 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:39:50.180356 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.171652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:36799 (local address 127.7.146.194:48489)
0619 06:39:50.172738 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:50.172758 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:50.172992 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:50.173532 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:50.173536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:50.173561 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:50.173931 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.173943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.175160 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.175165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.177570 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.177587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.179281 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.179289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.179550 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.179632 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 06:39:50.179733 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:39:50.179901 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":167,"thread_start_us":1180,"threads_started":1}
I20260619 06:39:50.180800 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.171677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:45073 (local address 127.7.146.194:39805)
0619 06:39:50.173248 (+ 1571us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:50.173265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:50.173389 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:50.173732 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:50.173736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:50.173755 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:50.174124 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.174136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.175134 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.175139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.177570 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.177587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.179281 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.179289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.179554 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.179629 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 06:39:50.179739 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 06:39:50.179901 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"mutex_wait_us":114,"thread_start_us":579,"threads_started":1}
I20260619 06:39:50.181371 9946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef23fbcc08949ef9fffa466400369db" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" is_pre_election: true
I20260619 06:39:50.181754 9667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef23fbcc08949ef9fffa466400369db" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2dafd092184cfda75e35989d449921" is_pre_election: true
I20260619 06:39:50.181994 9946 raft_consensus.cc:2454] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 0.
I20260619 06:39:50.182263 9667 raft_consensus.cc:2454] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 0.
I20260619 06:39:50.182768 9734 leader_election.cc:302] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24e737c9ecba4e98b1b74bc79d42a49f, 5ad615111bc34050a01200f47afbfae1; no voters:
I20260619 06:39:50.183158 10017 raft_consensus.cc:2781] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:39:50.183310 10017 raft_consensus.cc:482] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:39:50.183404 10017 raft_consensus.cc:3037] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:50.185710 10017 raft_consensus.cc:504] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } }
I20260619 06:39:50.186337 10017 leader_election.cc:288] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 election: Requested vote from peers 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799), de2dafd092184cfda75e35989d449921 (127.7.146.193:45073)
I20260619 06:39:50.186900 9946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef23fbcc08949ef9fffa466400369db" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e737c9ecba4e98b1b74bc79d42a49f"
I20260619 06:39:50.186900 9667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef23fbcc08949ef9fffa466400369db" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2dafd092184cfda75e35989d449921"
I20260619 06:39:50.187153 9667 raft_consensus.cc:3037] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:50.187153 9946 raft_consensus.cc:3037] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:50.190475 9946 raft_consensus.cc:2454] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 1.
I20260619 06:39:50.190475 9667 raft_consensus.cc:2454] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 1.
I20260619 06:39:50.190972 9735 leader_election.cc:302] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad615111bc34050a01200f47afbfae1, de2dafd092184cfda75e35989d449921; no voters:
I20260619 06:39:50.191313 10017 raft_consensus.cc:2781] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:50.191756 10017 raft_consensus.cc:686] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 1 LEADER]: Becoming Leader. State: Replica: 5ad615111bc34050a01200f47afbfae1, State: Running, Role: LEADER
I20260619 06:39:50.192247 10017 consensus_queue.cc:237] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } }
I20260619 06:39:50.197045 9506 catalog_manager.cc:5495] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ad615111bc34050a01200f47afbfae1 (127.7.146.194). New cstate: current_term: 1 leader_uuid: "5ad615111bc34050a01200f47afbfae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } health_report { overall_health: HEALTHY } } }
Jun 19 06:39:50 dist-test-slave-qfk4 krb5kdc[9461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851188, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:50.268172 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.255083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.7.146.254:33771)
0619 06:39:50.255284 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:50.255291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:50.255807 (+ 516us) server_negotiation.cc:407] Connection header received
0619 06:39:50.255973 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:50.255978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:50.256043 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:50.256121 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:50.257439 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.258130 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.259319 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.259727 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.259830 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.262926 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:50.262930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:50.262933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:50.262960 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:50.265539 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:50.266294 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:50.266299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:50.266301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:50.266415 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:50.266734 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:50.266739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:50.266743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:50.267023 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:50.267149 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:50.267544 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 06:39:50.267851 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":82}
I20260619 06:39:50.268178 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.255074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33771 (local address 127.0.0.1:54932)
0619 06:39:50.255658 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:50.255674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:50.255866 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:50.256171 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:50.256175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:50.257047 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:50.257294 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.257304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.258249 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.258254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.259191 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.259203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.259894 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.259899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.260038 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.260691 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:50.260715 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:50.262767 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:50.265674 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:50.265688 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:50.265697 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:50.266182 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:50.266512 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:50.266516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:50.266519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:50.266640 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:50.267140 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:50.267148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:50.267310 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 06:39:50.267445 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 06:39:50.267717 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":308,"threads_started":1}
I20260619 06:39:50.273416 9506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:54932:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:39:50.274369 9506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:39:50.280857 9643 tablet_service.cc:1467] Processing CreateTablet for tablet aadadfdcb60747cc8b6cc480ae830f6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71462d947bdc4e51847a5c2f01beb92a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:39:50.280982 9782 tablet_service.cc:1467] Processing CreateTablet for tablet aadadfdcb60747cc8b6cc480ae830f6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71462d947bdc4e51847a5c2f01beb92a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:39:50.281157 9921 tablet_service.cc:1467] Processing CreateTablet for tablet aadadfdcb60747cc8b6cc480ae830f6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71462d947bdc4e51847a5c2f01beb92a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:39:50.281795 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadadfdcb60747cc8b6cc480ae830f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:50.281904 9921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadadfdcb60747cc8b6cc480ae830f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:50.281904 9782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadadfdcb60747cc8b6cc480ae830f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:50.287050 10012 tablet_bootstrap.cc:492] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: Bootstrap starting.
I20260619 06:39:50.288309 10012 tablet_bootstrap.cc:654] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:50.288763 10011 tablet_bootstrap.cc:492] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1: Bootstrap starting.
I20260619 06:39:50.289729 10010 tablet_bootstrap.cc:492] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f: Bootstrap starting.
I20260619 06:39:50.290366 10011 tablet_bootstrap.cc:654] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:50.290663 10012 tablet_bootstrap.cc:492] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: No bootstrap required, opened a new log
I20260619 06:39:50.290789 10012 ts_tablet_manager.cc:1397] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 06:39:50.291290 10010 tablet_bootstrap.cc:654] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:50.291461 10012 raft_consensus.cc:348] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.291692 10012 raft_consensus.cc:374] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:50.291785 10012 raft_consensus.cc:729] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2dafd092184cfda75e35989d449921, State: Initialized, Role: FOLLOWER
I20260619 06:39:50.291999 10012 consensus_queue.cc:260] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.293028 10018 tablet_replica.cc:439] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } }
I20260619 06:39:50.293414 10018 tablet_replica.cc:442] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:50.293524 10010 tablet_bootstrap.cc:492] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f: No bootstrap required, opened a new log
I20260619 06:39:50.293653 10010 ts_tablet_manager.cc:1397] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260619 06:39:50.293850 10012 ts_tablet_manager.cc:1428] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 06:39:50.294318 10010 raft_consensus.cc:348] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.294535 10010 raft_consensus.cc:374] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:50.294626 10010 raft_consensus.cc:729] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e737c9ecba4e98b1b74bc79d42a49f, State: Initialized, Role: FOLLOWER
I20260619 06:39:50.294858 10010 consensus_queue.cc:260] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.295037 10011 tablet_bootstrap.cc:492] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1: No bootstrap required, opened a new log
I20260619 06:39:50.295185 10011 ts_tablet_manager.cc:1397] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 06:39:50.295342 10010 ts_tablet_manager.cc:1428] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 06:39:50.295636 10016 tablet_replica.cc:439] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } }
I20260619 06:39:50.295979 10016 tablet_replica.cc:442] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:50.295933 10011 raft_consensus.cc:348] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.296159 10011 raft_consensus.cc:374] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:50.296257 10011 raft_consensus.cc:729] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad615111bc34050a01200f47afbfae1, State: Initialized, Role: FOLLOWER
I20260619 06:39:50.296473 10011 consensus_queue.cc:260] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.297137 10011 ts_tablet_manager.cc:1428] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:39:50.296990 10017 tablet_replica.cc:439] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } }
I20260619 06:39:50.297277 10017 tablet_replica.cc:442] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:50.319754 10018 raft_consensus.cc:482] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:39:50.320029 10018 raft_consensus.cc:504] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.321820 10018 leader_election.cc:288] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ad615111bc34050a01200f47afbfae1 (127.7.146.194:39145), 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799)
I20260619 06:39:50.328897 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.321957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:39145 (local address 127.7.146.193:35297)
0619 06:39:50.323019 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:50.323042 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:50.323223 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:50.323564 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:50.323570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:50.323598 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:50.324093 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.324104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.325164 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.325170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.326917 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.326930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.328078 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.328084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.328269 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.328302 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:39:50.328382 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:39:50.328477 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":27,"thread_start_us":719,"threads_started":1}
I20260619 06:39:50.328914 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.322235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:35297 (local address 127.7.146.194:39145)
0619 06:39:50.322433 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:50.322441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:50.323151 (+ 710us) server_negotiation.cc:407] Connection header received
0619 06:39:50.323370 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:50.323377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:50.323454 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:50.323528 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:50.324276 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.325042 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.327074 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.327855 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.327979 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.328033 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:50.328438 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 06:39:50.328600 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:39:50.328991 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.322716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:57729 (local address 127.7.146.195:36799)
0619 06:39:50.322896 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:50.322901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:50.323413 (+ 512us) server_negotiation.cc:407] Connection header received
0619 06:39:50.323564 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:50.323570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:50.323646 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:50.323740 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:39:50.324315 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.325185 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.327046 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.328056 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.328220 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.328287 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:50.328611 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 06:39:50.328761 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:39:50.329049 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.322359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:36799 (local address 127.7.146.193:57729)
0619 06:39:50.323304 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:50.323320 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:50.323442 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:50.323777 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:50.323780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:50.323812 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:39:50.324143 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.324152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.325319 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.325325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.326885 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.326900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.328285 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.328289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.328431 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.328461 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:39:50.328544 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 06:39:50.328700 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":23,"thread_start_us":636,"threads_started":1}
I20260619 06:39:50.330036 9946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadadfdcb60747cc8b6cc480ae830f6e" candidate_uuid: "de2dafd092184cfda75e35989d449921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" is_pre_election: true
I20260619 06:39:50.330320 9946 raft_consensus.cc:2454] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2dafd092184cfda75e35989d449921 in term 0.
I20260619 06:39:50.330288 9801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadadfdcb60747cc8b6cc480ae830f6e" candidate_uuid: "de2dafd092184cfda75e35989d449921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad615111bc34050a01200f47afbfae1" is_pre_election: true
I20260619 06:39:50.330672 9801 raft_consensus.cc:2454] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2dafd092184cfda75e35989d449921 in term 0.
I20260619 06:39:50.330765 9594 leader_election.cc:302] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24e737c9ecba4e98b1b74bc79d42a49f, de2dafd092184cfda75e35989d449921; no voters:
I20260619 06:39:50.331154 10018 raft_consensus.cc:2781] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:39:50.331274 10018 raft_consensus.cc:482] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:39:50.331368 10018 raft_consensus.cc:3037] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:50.333449 10018 raft_consensus.cc:504] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.334005 10018 leader_election.cc:288] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [CANDIDATE]: Term 1 election: Requested vote from peers 5ad615111bc34050a01200f47afbfae1 (127.7.146.194:39145), 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799)
I20260619 06:39:50.334486 9946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadadfdcb60747cc8b6cc480ae830f6e" candidate_uuid: "de2dafd092184cfda75e35989d449921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e737c9ecba4e98b1b74bc79d42a49f"
I20260619 06:39:50.334486 9801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadadfdcb60747cc8b6cc480ae830f6e" candidate_uuid: "de2dafd092184cfda75e35989d449921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad615111bc34050a01200f47afbfae1"
I20260619 06:39:50.334656 9801 raft_consensus.cc:3037] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:50.334667 9946 raft_consensus.cc:3037] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:50.336709 9946 raft_consensus.cc:2454] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2dafd092184cfda75e35989d449921 in term 1.
I20260619 06:39:50.336946 9801 raft_consensus.cc:2454] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2dafd092184cfda75e35989d449921 in term 1.
I20260619 06:39:50.337096 9594 leader_election.cc:302] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24e737c9ecba4e98b1b74bc79d42a49f, de2dafd092184cfda75e35989d449921; no voters:
I20260619 06:39:50.337383 10018 raft_consensus.cc:2781] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:50.337862 10018 raft_consensus.cc:686] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [term 1 LEADER]: Becoming Leader. State: Replica: de2dafd092184cfda75e35989d449921, State: Running, Role: LEADER
I20260619 06:39:50.338306 10018 consensus_queue.cc:237] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.341502 10034 tablet_replica.cc:439] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: TxnStatusTablet state changed. Reason: New leader de2dafd092184cfda75e35989d449921. Latest consensus state: current_term: 1 leader_uuid: "de2dafd092184cfda75e35989d449921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } }
I20260619 06:39:50.341881 10034 tablet_replica.cc:442] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:39:50.342623 10036 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:39:50.342887 10036 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:39:50.343467 9503 catalog_manager.cc:5495] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 reported cstate change: term changed from 0 to 1, leader changed from <none> to de2dafd092184cfda75e35989d449921 (127.7.146.193). New cstate: current_term: 1 leader_uuid: "de2dafd092184cfda75e35989d449921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } health_report { overall_health: UNKNOWN } } }
I20260619 06:39:50.370708 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.364513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:45073 (local address 127.0.0.1:33238)
0619 06:39:50.364759 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:50.364778 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:50.364895 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:50.365361 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:50.365364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:50.365374 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:39:50.365521 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.365527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.366902 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.366905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.368076 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.368088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.369047 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.369052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.369185 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.369253 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:39:50.370242 (+ 989us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:39:50.370261 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:39:50.370359 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 06:39:50.370477 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 06:39:50.370860 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.364556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.7.146.193:45073)
0619 06:39:50.364775 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:50.364780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:50.364844 (+ 64us) server_negotiation.cc:407] Connection header received
0619 06:39:50.365070 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:50.365076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:50.365180 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:50.365345 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:50.365640 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.366787 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.368215 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.368893 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.369017 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.369427 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:50.370119 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:50.370260 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:50.370400 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 06:39:50.370483 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 06:39:50.379148 9946 raft_consensus.cc:1261] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Refusing update from remote peer de2dafd092184cfda75e35989d449921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:50.379180 9801 raft_consensus.cc:1261] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Refusing update from remote peer de2dafd092184cfda75e35989d449921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:50.380038 10034 consensus_queue.cc:1035] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:50.380467 10018 consensus_queue.cc:1035] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:50.390611 10016 tablet_replica.cc:439] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f: TxnStatusTablet state changed. Reason: New leader de2dafd092184cfda75e35989d449921. Latest consensus state: current_term: 1 leader_uuid: "de2dafd092184cfda75e35989d449921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } }
I20260619 06:39:50.390893 10016 tablet_replica.cc:442] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:50.391590 10017 tablet_replica.cc:439] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1: TxnStatusTablet state changed. Reason: New leader de2dafd092184cfda75e35989d449921. Latest consensus state: current_term: 1 leader_uuid: "de2dafd092184cfda75e35989d449921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } }
I20260619 06:39:50.392136 10017 tablet_replica.cc:442] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:50.394124 10018 tablet_replica.cc:439] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de2dafd092184cfda75e35989d449921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } }
I20260619 06:39:50.394423 10018 tablet_replica.cc:442] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:39:50.405656 10034 tablet_replica.cc:439] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de2dafd092184cfda75e35989d449921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } }
I20260619 06:39:50.405923 10034 tablet_replica.cc:442] T aadadfdcb60747cc8b6cc480ae830f6e P de2dafd092184cfda75e35989d449921: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:39:50.407881 10040 mvcc.cc:205] Tried to move back new op lower bound from 7298462475783598080 to 7298462475629113344. Current Snapshot: MvccSnapshot[applied={T|T < 7298462475783598080}]
I20260619 06:39:50.411369 10017 tablet_replica.cc:439] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de2dafd092184cfda75e35989d449921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } }
I20260619 06:39:50.411661 10017 tablet_replica.cc:442] T aadadfdcb60747cc8b6cc480ae830f6e P 5ad615111bc34050a01200f47afbfae1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:50.420795 10016 tablet_replica.cc:439] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de2dafd092184cfda75e35989d449921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } } }
I20260619 06:39:50.421073 10016 tablet_replica.cc:442] T aadadfdcb60747cc8b6cc480ae830f6e P 24e737c9ecba4e98b1b74bc79d42a49f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:39:50.424467 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.417466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.7.146.194:39145)
0619 06:39:50.417647 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:50.417654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:50.417701 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:39:50.417814 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:50.417820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:50.417919 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:50.418075 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:50.418485 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.419733 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.421769 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.422406 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.422526 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.423130 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:50.423812 (+ 682us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:50.423936 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:50.424084 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 06:39:50.424166 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:39:50.441520 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.435594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:43639 (local address 127.7.146.193:45073)
0619 06:39:50.435829 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:50.435836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:50.436372 (+ 536us) server_negotiation.cc:407] Connection header received
0619 06:39:50.436645 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:50.436651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:50.436740 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:50.436824 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:50.437326 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.438074 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.439447 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.439992 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.440107 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.440732 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:50.440861 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:50.440945 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:50.441091 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 06:39:50.441174 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:39:50.441550 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.435577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:45073 (local address 127.7.146.194:43639)
0619 06:39:50.436264 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:50.436279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:50.436437 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:50.436839 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:50.436844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:50.436855 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:39:50.437181 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.437191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.438200 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.438206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.439269 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.439281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.440155 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.440162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.440291 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.440345 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:39:50.440971 (+ 626us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:39:50.440982 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:39:50.441052 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:39:50.441191 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":380,"threads_started":1}
I20260619 06:39:50.453397 9946 raft_consensus.cc:1261] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Refusing update from remote peer 5ad615111bc34050a01200f47afbfae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:50.453540 9667 raft_consensus.cc:1261] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Refusing update from remote peer 5ad615111bc34050a01200f47afbfae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:50.454033 10021 consensus_queue.cc:1035] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:50.454704 10021 consensus_queue.cc:1035] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:50.463917 10039 mvcc.cc:205] Tried to move back new op lower bound from 7298462476090322944 to 7298462475031719936. Current Snapshot: MvccSnapshot[applied={T|T < 7298462476090322944}]
I20260619 06:39:50.496340 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.490102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:52513 (local address 127.7.146.194:39145)
0619 06:39:50.490300 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:50.490307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:50.490874 (+ 567us) server_negotiation.cc:407] Connection header received
0619 06:39:50.491058 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:50.491063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:50.491155 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:50.491244 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:50.491816 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.492913 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.494365 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.494954 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.495076 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.495598 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:50.495726 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:50.495809 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:50.495998 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 06:39:50.496072 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:39:50.496495 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:50.490024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:39145 (local address 127.7.146.193:52513)
0619 06:39:50.490769 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:50.490789 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:50.490934 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:50.491283 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:50.491288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:50.491300 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:39:50.491649 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.491658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.493035 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.493038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.494198 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:50.494214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:50.495153 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:50.495160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:50.495351 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:50.495419 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:39:50.495856 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:39:50.495869 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 06:39:50.495971 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 06:39:50.496100 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":431,"threads_started":1}
I20260619 06:39:50.534137 9503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:50.534442 9503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:50.538632 9503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84: Sending DeleteTablet for 3 replicas of tablet 2ef23fbcc08949ef9fffa466400369db
I20260619 06:39:50.539808 9921 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef23fbcc08949ef9fffa466400369db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:50 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:44558
I20260619 06:39:50.540997 9782 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef23fbcc08949ef9fffa466400369db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:50 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:56984
I20260619 06:39:50.541028 9643 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef23fbcc08949ef9fffa466400369db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:50 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:33224
I20260619 06:39:50.542564 9506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:39:50.543169 9506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:39:50.545416 10059 tablet_replica.cc:330] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: stopping tablet replica
I20260619 06:39:50.552263 9781 tablet_service.cc:1467] Processing CreateTablet for tablet f4769ca21d9141b69956dc110ae50203 (DEFAULT_TABLE table=test-table [id=1eb34a1e41e14b5e9aa11793c71fc303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:50.552883 9781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4769ca21d9141b69956dc110ae50203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:50.553871 9921 tablet_service.cc:1467] Processing CreateTablet for tablet f4769ca21d9141b69956dc110ae50203 (DEFAULT_TABLE table=test-table [id=1eb34a1e41e14b5e9aa11793c71fc303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:50.554280 9921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4769ca21d9141b69956dc110ae50203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:50.557144 9642 tablet_service.cc:1467] Processing CreateTablet for tablet f4769ca21d9141b69956dc110ae50203 (DEFAULT_TABLE table=test-table [id=1eb34a1e41e14b5e9aa11793c71fc303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:50.557682 9642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4769ca21d9141b69956dc110ae50203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:50.562958 10010 tablet_bootstrap.cc:492] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f: Bootstrap starting.
I20260619 06:39:50.564538 10010 tablet_bootstrap.cc:654] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:50.546007 10059 raft_consensus.cc:2229] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:50.567590 10010 tablet_bootstrap.cc:492] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f: No bootstrap required, opened a new log
I20260619 06:39:50.567766 10011 tablet_bootstrap.cc:492] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1: Bootstrap starting.
I20260619 06:39:50.567795 10010 ts_tablet_manager.cc:1397] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:39:50.568528 10010 raft_consensus.cc:348] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.568832 10010 raft_consensus.cc:374] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:50.568913 10010 raft_consensus.cc:729] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e737c9ecba4e98b1b74bc79d42a49f, State: Initialized, Role: FOLLOWER
I20260619 06:39:50.569350 10011 tablet_bootstrap.cc:654] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:50.569253 10010 consensus_queue.cc:260] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.572364 10010 ts_tablet_manager.cc:1428] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:39:50.574357 10012 tablet_bootstrap.cc:492] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921: Bootstrap starting.
I20260619 06:39:50.567600 10059 raft_consensus.cc:2258] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:50.576174 10012 tablet_bootstrap.cc:654] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:50.577196 10059 ts_tablet_manager.cc:1905] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:50.586201 10059 ts_tablet_manager.cc:1918] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:39:50.586386 10059 log.cc:1196] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/2ef23fbcc08949ef9fffa466400369db
I20260619 06:39:50.586898 10059 ts_tablet_manager.cc:1939] T 2ef23fbcc08949ef9fffa466400369db P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting consensus metadata
I20260619 06:39:50.590167 9491 catalog_manager.cc:4841] TS 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799): tablet 2ef23fbcc08949ef9fffa466400369db (table test-table [id=38ed70a2d136424e865522d046342b1c]) successfully deleted
I20260619 06:39:50.596421 10011 tablet_bootstrap.cc:492] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1: No bootstrap required, opened a new log
I20260619 06:39:50.596432 10012 tablet_bootstrap.cc:492] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921: No bootstrap required, opened a new log
I20260619 06:39:50.596642 10012 ts_tablet_manager.cc:1397] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.003s
I20260619 06:39:50.597469 10012 raft_consensus.cc:348] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.597733 10012 raft_consensus.cc:374] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:50.597827 10012 raft_consensus.cc:729] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2dafd092184cfda75e35989d449921, State: Initialized, Role: FOLLOWER
I20260619 06:39:50.597903 10011 ts_tablet_manager.cc:1397] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.005s
I20260619 06:39:50.598752 10011 raft_consensus.cc:348] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.599006 10011 raft_consensus.cc:374] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:50.599092 10011 raft_consensus.cc:729] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad615111bc34050a01200f47afbfae1, State: Initialized, Role: FOLLOWER
I20260619 06:39:50.599399 10011 consensus_queue.cc:260] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.600039 10011 ts_tablet_manager.cc:1428] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:39:50.600289 10012 consensus_queue.cc:260] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.601821 10012 ts_tablet_manager.cc:1428] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260619 06:39:50.603693 10060 tablet_replica.cc:330] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: stopping tablet replica
I20260619 06:39:50.604192 10061 tablet_replica.cc:330] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: stopping tablet replica
I20260619 06:39:50.604660 10061 raft_consensus.cc:2229] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:39:50.605298 10060 raft_consensus.cc:2229] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:50.605616 10060 raft_consensus.cc:2258] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:50.606102 10061 raft_consensus.cc:2258] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:50.609157 10061 ts_tablet_manager.cc:1905] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:50.609350 10060 ts_tablet_manager.cc:1905] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:50.617102 10061 ts_tablet_manager.cc:1918] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:39:50.617199 10060 ts_tablet_manager.cc:1918] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:39:50.617417 10061 log.cc:1196] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/2ef23fbcc08949ef9fffa466400369db
I20260619 06:39:50.617374 10060 log.cc:1196] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/2ef23fbcc08949ef9fffa466400369db
I20260619 06:39:50.617973 10060 ts_tablet_manager.cc:1939] T 2ef23fbcc08949ef9fffa466400369db P de2dafd092184cfda75e35989d449921: Deleting consensus metadata
I20260619 06:39:50.618089 10061 ts_tablet_manager.cc:1939] T 2ef23fbcc08949ef9fffa466400369db P 5ad615111bc34050a01200f47afbfae1: Deleting consensus metadata
I20260619 06:39:50.619633 9490 catalog_manager.cc:4841] TS 5ad615111bc34050a01200f47afbfae1 (127.7.146.194:39145): tablet 2ef23fbcc08949ef9fffa466400369db (table test-table [id=38ed70a2d136424e865522d046342b1c]) successfully deleted
I20260619 06:39:50.619825 9492 catalog_manager.cc:4841] TS de2dafd092184cfda75e35989d449921 (127.7.146.193:45073): tablet 2ef23fbcc08949ef9fffa466400369db (table test-table [id=38ed70a2d136424e865522d046342b1c]) successfully deleted
I20260619 06:39:50.656016 10053 raft_consensus.cc:482] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:39:50.656268 10053 raft_consensus.cc:504] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.657115 10053 leader_election.cc:288] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2dafd092184cfda75e35989d449921 (127.7.146.193:45073), 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799)
I20260619 06:39:50.657876 9667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4769ca21d9141b69956dc110ae50203" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2dafd092184cfda75e35989d449921" is_pre_election: true
I20260619 06:39:50.658144 9667 raft_consensus.cc:2454] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 0.
I20260619 06:39:50.658572 9735 leader_election.cc:302] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad615111bc34050a01200f47afbfae1, de2dafd092184cfda75e35989d449921; no voters:
I20260619 06:39:50.658521 9946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4769ca21d9141b69956dc110ae50203" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" is_pre_election: true
I20260619 06:39:50.658759 9946 raft_consensus.cc:2454] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 0.
I20260619 06:39:50.658849 10053 raft_consensus.cc:2781] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:39:50.658951 10053 raft_consensus.cc:482] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:39:50.659041 10053 raft_consensus.cc:3037] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:50.662196 10053 raft_consensus.cc:504] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.663370 9667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4769ca21d9141b69956dc110ae50203" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2dafd092184cfda75e35989d449921"
I20260619 06:39:50.663378 9946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4769ca21d9141b69956dc110ae50203" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e737c9ecba4e98b1b74bc79d42a49f"
I20260619 06:39:50.663582 9946 raft_consensus.cc:3037] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:50.663596 9667 raft_consensus.cc:3037] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:50.666534 9946 raft_consensus.cc:2454] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 1.
I20260619 06:39:50.666788 9667 raft_consensus.cc:2454] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 1.
I20260619 06:39:50.667114 9734 leader_election.cc:302] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24e737c9ecba4e98b1b74bc79d42a49f, 5ad615111bc34050a01200f47afbfae1; no voters:
I20260619 06:39:50.667448 10017 raft_consensus.cc:2781] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:50.667598 10053 leader_election.cc:288] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 election: Requested vote from peers de2dafd092184cfda75e35989d449921 (127.7.146.193:45073), 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799)
I20260619 06:39:50.667620 10017 raft_consensus.cc:686] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 1 LEADER]: Becoming Leader. State: Replica: 5ad615111bc34050a01200f47afbfae1, State: Running, Role: LEADER
I20260619 06:39:50.668277 10017 consensus_queue.cc:237] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:50.671404 9503 catalog_manager.cc:5495] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ad615111bc34050a01200f47afbfae1 (127.7.146.194). New cstate: current_term: 1 leader_uuid: "5ad615111bc34050a01200f47afbfae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } health_report { overall_health: UNKNOWN } } }
I20260619 06:39:50.709160 9667 raft_consensus.cc:1261] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Refusing update from remote peer 5ad615111bc34050a01200f47afbfae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:50.709336 9946 raft_consensus.cc:1261] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Refusing update from remote peer 5ad615111bc34050a01200f47afbfae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:50.709923 10017 consensus_queue.cc:1035] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:50.710233 10053 consensus_queue.cc:1035] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:50.728667 10041 mvcc.cc:205] Tried to move back new op lower bound from 7298462477139210240 to 7298462476980563968. Current Snapshot: MvccSnapshot[applied={T|T < 7298462477139210240}]
W20260619 06:39:50.754699 10040 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:39:52.353904 9504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:52.354122 9504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:52.358261 9504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84: Sending DeleteTablet for 3 replicas of tablet f4769ca21d9141b69956dc110ae50203
I20260619 06:39:52.359099 9782 tablet_service.cc:1514] Processing DeleteTablet for tablet f4769ca21d9141b69956dc110ae50203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:52 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:56984
I20260619 06:39:52.359205 9921 tablet_service.cc:1514] Processing DeleteTablet for tablet f4769ca21d9141b69956dc110ae50203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:52 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:44558
I20260619 06:39:52.359323 9643 tablet_service.cc:1514] Processing DeleteTablet for tablet f4769ca21d9141b69956dc110ae50203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:52 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:33224
I20260619 06:39:52.370843 10092 tablet_replica.cc:330] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921: stopping tablet replica
I20260619 06:39:52.371089 10092 raft_consensus.cc:2229] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:52.371382 10091 tablet_replica.cc:330] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f: stopping tablet replica
I20260619 06:39:52.371570 10091 raft_consensus.cc:2229] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:52.371707 10092 raft_consensus.cc:2258] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:52.371729 10090 tablet_replica.cc:330] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1: stopping tablet replica
I20260619 06:39:52.371802 10091 raft_consensus.cc:2258] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:52.371991 10090 raft_consensus.cc:2229] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:39:52.372386 10090 raft_consensus.cc:2258] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:52.372823 10091 ts_tablet_manager.cc:1905] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:52.373147 10092 ts_tablet_manager.cc:1905] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:52.373436 10090 ts_tablet_manager.cc:1905] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:52.379449 10092 ts_tablet_manager.cc:1918] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:39:52.379464 10091 ts_tablet_manager.cc:1918] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:39:52.379602 10091 log.cc:1196] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/f4769ca21d9141b69956dc110ae50203
I20260619 06:39:52.379607 10092 log.cc:1196] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/f4769ca21d9141b69956dc110ae50203
I20260619 06:39:52.379753 10090 ts_tablet_manager.cc:1918] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:39:52.379918 10090 log.cc:1196] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/f4769ca21d9141b69956dc110ae50203
I20260619 06:39:52.379988 10092 ts_tablet_manager.cc:1939] T f4769ca21d9141b69956dc110ae50203 P de2dafd092184cfda75e35989d449921: Deleting consensus metadata
I20260619 06:39:52.380012 10091 ts_tablet_manager.cc:1939] T f4769ca21d9141b69956dc110ae50203 P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting consensus metadata
I20260619 06:39:52.380405 10090 ts_tablet_manager.cc:1939] T f4769ca21d9141b69956dc110ae50203 P 5ad615111bc34050a01200f47afbfae1: Deleting consensus metadata
I20260619 06:39:52.381155 9491 catalog_manager.cc:4841] TS 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799): tablet f4769ca21d9141b69956dc110ae50203 (table test-table [id=1eb34a1e41e14b5e9aa11793c71fc303]) successfully deleted
I20260619 06:39:52.381155 9492 catalog_manager.cc:4841] TS de2dafd092184cfda75e35989d449921 (127.7.146.193:45073): tablet f4769ca21d9141b69956dc110ae50203 (table test-table [id=1eb34a1e41e14b5e9aa11793c71fc303]) successfully deleted
I20260619 06:39:52.381502 9490 catalog_manager.cc:4841] TS 5ad615111bc34050a01200f47afbfae1 (127.7.146.194:39145): tablet f4769ca21d9141b69956dc110ae50203 (table test-table [id=1eb34a1e41e14b5e9aa11793c71fc303]) successfully deleted
Jun 19 06:39:52 dist-test-slave-qfk4 krb5kdc[9461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:39:52 dist-test-slave-qfk4 krb5kdc[9461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:52.419241 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:52.403671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.7.146.254:33771)
0619 06:39:52.404476 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:52.404480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:52.404500 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:39:52.404552 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:52.404555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:52.404662 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:52.404750 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:52.406401 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:52.407177 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:52.409037 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:52.409541 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:52.409655 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:52.414177 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:52.414183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:52.414186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:52.414223 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:52.416977 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:52.417664 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:52.417669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:52.417671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:52.417789 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:52.418089 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:52.418092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:52.418094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:52.418379 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:52.418553 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:52.418760 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 06:39:52.418943 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":569,"threads_started":1}
I20260619 06:39:52.422673 9504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:39:52.423305 9504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:39:52.429562 9643 tablet_service.cc:1467] Processing CreateTablet for tablet 6deb58f9a0754d11a8384b8c36aa00d1 (DEFAULT_TABLE table=test-table [id=1617189537f74630972929e1d5a3de53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:52.429648 9782 tablet_service.cc:1467] Processing CreateTablet for tablet 6deb58f9a0754d11a8384b8c36aa00d1 (DEFAULT_TABLE table=test-table [id=1617189537f74630972929e1d5a3de53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:52.429877 9921 tablet_service.cc:1467] Processing CreateTablet for tablet 6deb58f9a0754d11a8384b8c36aa00d1 (DEFAULT_TABLE table=test-table [id=1617189537f74630972929e1d5a3de53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:52.430022 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6deb58f9a0754d11a8384b8c36aa00d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:52.430063 9782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6deb58f9a0754d11a8384b8c36aa00d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:52.430246 9921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6deb58f9a0754d11a8384b8c36aa00d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:52.436038 10101 tablet_bootstrap.cc:492] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1: Bootstrap starting.
I20260619 06:39:52.437031 10102 tablet_bootstrap.cc:492] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921: Bootstrap starting.
I20260619 06:39:52.437919 10101 tablet_bootstrap.cc:654] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:52.438665 10102 tablet_bootstrap.cc:654] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:52.439167 10103 tablet_bootstrap.cc:492] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f: Bootstrap starting.
I20260619 06:39:52.440083 10101 tablet_bootstrap.cc:492] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1: No bootstrap required, opened a new log
I20260619 06:39:52.440292 10101 ts_tablet_manager.cc:1397] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260619 06:39:52.440744 10103 tablet_bootstrap.cc:654] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:52.440865 10102 tablet_bootstrap.cc:492] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921: No bootstrap required, opened a new log
I20260619 06:39:52.440999 10102 ts_tablet_manager.cc:1397] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 06:39:52.441102 10101 raft_consensus.cc:348] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.441313 10101 raft_consensus.cc:374] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:52.441407 10101 raft_consensus.cc:729] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad615111bc34050a01200f47afbfae1, State: Initialized, Role: FOLLOWER
I20260619 06:39:52.441634 10101 consensus_queue.cc:260] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.441784 10102 raft_consensus.cc:348] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.441996 10102 raft_consensus.cc:374] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:52.442093 10102 raft_consensus.cc:729] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2dafd092184cfda75e35989d449921, State: Initialized, Role: FOLLOWER
I20260619 06:39:52.442516 10101 ts_tablet_manager.cc:1428] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:39:52.442341 10102 consensus_queue.cc:260] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.443092 10103 tablet_bootstrap.cc:492] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f: No bootstrap required, opened a new log
I20260619 06:39:52.443228 10103 ts_tablet_manager.cc:1397] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260619 06:39:52.443759 10102 ts_tablet_manager.cc:1428] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:39:52.443959 10103 raft_consensus.cc:348] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.444173 10103 raft_consensus.cc:374] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:52.444264 10103 raft_consensus.cc:729] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e737c9ecba4e98b1b74bc79d42a49f, State: Initialized, Role: FOLLOWER
I20260619 06:39:52.444473 10103 consensus_queue.cc:260] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.445957 10103 ts_tablet_manager.cc:1428] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260619 06:39:52.454294 10081 raft_consensus.cc:482] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:39:52.454492 10081 raft_consensus.cc:504] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.455127 10081 leader_election.cc:288] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2dafd092184cfda75e35989d449921 (127.7.146.193:45073), 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799)
I20260619 06:39:52.455651 9946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6deb58f9a0754d11a8384b8c36aa00d1" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" is_pre_election: true
I20260619 06:39:52.455616 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6deb58f9a0754d11a8384b8c36aa00d1" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2dafd092184cfda75e35989d449921" is_pre_election: true
I20260619 06:39:52.455874 9665 raft_consensus.cc:2454] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 0.
I20260619 06:39:52.455930 9946 raft_consensus.cc:2454] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 0.
I20260619 06:39:52.456377 9735 leader_election.cc:302] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad615111bc34050a01200f47afbfae1, de2dafd092184cfda75e35989d449921; no voters:
I20260619 06:39:52.456646 10081 raft_consensus.cc:2781] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:39:52.456765 10081 raft_consensus.cc:482] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:39:52.456858 10081 raft_consensus.cc:3037] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:52.458779 10081 raft_consensus.cc:504] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.459293 10081 leader_election.cc:288] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 election: Requested vote from peers de2dafd092184cfda75e35989d449921 (127.7.146.193:45073), 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799)
I20260619 06:39:52.459757 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6deb58f9a0754d11a8384b8c36aa00d1" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2dafd092184cfda75e35989d449921"
I20260619 06:39:52.459826 9946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6deb58f9a0754d11a8384b8c36aa00d1" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e737c9ecba4e98b1b74bc79d42a49f"
I20260619 06:39:52.459941 9665 raft_consensus.cc:3037] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:52.460040 9946 raft_consensus.cc:3037] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:52.462354 9665 raft_consensus.cc:2454] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 1.
I20260619 06:39:52.462369 9946 raft_consensus.cc:2454] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 1.
I20260619 06:39:52.462719 9734 leader_election.cc:302] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24e737c9ecba4e98b1b74bc79d42a49f, 5ad615111bc34050a01200f47afbfae1; no voters:
I20260619 06:39:52.462985 10081 raft_consensus.cc:2781] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:52.463129 10081 raft_consensus.cc:686] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 1 LEADER]: Becoming Leader. State: Replica: 5ad615111bc34050a01200f47afbfae1, State: Running, Role: LEADER
I20260619 06:39:52.463364 10081 consensus_queue.cc:237] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.465879 9503 catalog_manager.cc:5495] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ad615111bc34050a01200f47afbfae1 (127.7.146.194). New cstate: current_term: 1 leader_uuid: "5ad615111bc34050a01200f47afbfae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } health_report { overall_health: UNKNOWN } } }
I20260619 06:39:52.487216 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:52.482426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.7.146.194:39145)
0619 06:39:52.482916 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:52.482921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:52.482940 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:39:52.482990 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:52.482994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:52.483054 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:52.483148 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:39:52.483499 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:52.484202 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:52.485556 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:52.485955 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:52.486048 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:52.486517 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:39:52.486629 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:39:52.486707 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:52.486836 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 06:39:52.486900 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":287,"threads_started":1}
I20260619 06:39:52.495644 9946 raft_consensus.cc:1261] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Refusing update from remote peer 5ad615111bc34050a01200f47afbfae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:52.495647 9665 raft_consensus.cc:1261] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Refusing update from remote peer 5ad615111bc34050a01200f47afbfae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:52.496192 10084 consensus_queue.cc:1035] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:52.496440 10081 consensus_queue.cc:1035] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:52.536541 9504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:52.536830 9504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:52.539523 9504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84: Sending DeleteTablet for 3 replicas of tablet 6deb58f9a0754d11a8384b8c36aa00d1
I20260619 06:39:52.540316 9643 tablet_service.cc:1514] Processing DeleteTablet for tablet 6deb58f9a0754d11a8384b8c36aa00d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:52 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:33224
I20260619 06:39:52.540364 9782 tablet_service.cc:1514] Processing DeleteTablet for tablet 6deb58f9a0754d11a8384b8c36aa00d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:52 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:56984
I20260619 06:39:52.540395 9921 tablet_service.cc:1514] Processing DeleteTablet for tablet 6deb58f9a0754d11a8384b8c36aa00d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:52 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:44558
I20260619 06:39:52.540678 10090 tablet_replica.cc:330] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1: stopping tablet replica
I20260619 06:39:52.540690 10092 tablet_replica.cc:330] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921: stopping tablet replica
I20260619 06:39:52.540700 10091 tablet_replica.cc:330] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f: stopping tablet replica
I20260619 06:39:52.540869 10090 raft_consensus.cc:2229] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:39:52.540874 10092 raft_consensus.cc:2229] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:52.540884 10091 raft_consensus.cc:2229] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:52.541035 10092 raft_consensus.cc:2258] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:52.541066 10091 raft_consensus.cc:2258] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:52.541160 10090 raft_consensus.cc:2258] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:52.541785 9504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:39:52.542119 10090 ts_tablet_manager.cc:1905] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:52.542119 10092 ts_tablet_manager.cc:1905] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:52.542152 10091 ts_tablet_manager.cc:1905] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 06:39:52.542477 9504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:39:52.548237 10091 ts_tablet_manager.cc:1918] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:39:52.548692 10092 ts_tablet_manager.cc:1918] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:39:52.548899 10092 log.cc:1196] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/6deb58f9a0754d11a8384b8c36aa00d1
I20260619 06:39:52.548923 10090 ts_tablet_manager.cc:1918] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:39:52.549111 10090 log.cc:1196] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/6deb58f9a0754d11a8384b8c36aa00d1
I20260619 06:39:52.549309 10092 ts_tablet_manager.cc:1939] T 6deb58f9a0754d11a8384b8c36aa00d1 P de2dafd092184cfda75e35989d449921: Deleting consensus metadata
I20260619 06:39:52.549453 10090 ts_tablet_manager.cc:1939] T 6deb58f9a0754d11a8384b8c36aa00d1 P 5ad615111bc34050a01200f47afbfae1: Deleting consensus metadata
I20260619 06:39:52.548914 10091 log.cc:1196] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/6deb58f9a0754d11a8384b8c36aa00d1
I20260619 06:39:52.549993 10091 ts_tablet_manager.cc:1939] T 6deb58f9a0754d11a8384b8c36aa00d1 P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting consensus metadata
I20260619 06:39:52.550442 9490 catalog_manager.cc:4841] TS 5ad615111bc34050a01200f47afbfae1 (127.7.146.194:39145): tablet 6deb58f9a0754d11a8384b8c36aa00d1 (table test-table [id=1617189537f74630972929e1d5a3de53]) successfully deleted
I20260619 06:39:52.550781 9492 catalog_manager.cc:4841] TS de2dafd092184cfda75e35989d449921 (127.7.146.193:45073): tablet 6deb58f9a0754d11a8384b8c36aa00d1 (table test-table [id=1617189537f74630972929e1d5a3de53]) successfully deleted
I20260619 06:39:52.551052 9491 catalog_manager.cc:4841] TS 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799): tablet 6deb58f9a0754d11a8384b8c36aa00d1 (table test-table [id=1617189537f74630972929e1d5a3de53]) successfully deleted
I20260619 06:39:52.553377 9643 tablet_service.cc:1467] Processing CreateTablet for tablet e9a65279071c4211aede27de9bc109a6 (DEFAULT_TABLE table=test-table [id=735fb4f08efa424883d7932973a54637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:52.553545 9782 tablet_service.cc:1467] Processing CreateTablet for tablet e9a65279071c4211aede27de9bc109a6 (DEFAULT_TABLE table=test-table [id=735fb4f08efa424883d7932973a54637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:52.553575 9921 tablet_service.cc:1467] Processing CreateTablet for tablet e9a65279071c4211aede27de9bc109a6 (DEFAULT_TABLE table=test-table [id=735fb4f08efa424883d7932973a54637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:39:52.553781 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a65279071c4211aede27de9bc109a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:52.553929 9921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a65279071c4211aede27de9bc109a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:52.553930 9782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a65279071c4211aede27de9bc109a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:52.557960 10102 tablet_bootstrap.cc:492] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921: Bootstrap starting.
I20260619 06:39:52.559417 10102 tablet_bootstrap.cc:654] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:52.559827 10101 tablet_bootstrap.cc:492] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1: Bootstrap starting.
I20260619 06:39:52.559885 10103 tablet_bootstrap.cc:492] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f: Bootstrap starting.
I20260619 06:39:52.561267 10101 tablet_bootstrap.cc:654] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:52.561374 10102 tablet_bootstrap.cc:492] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921: No bootstrap required, opened a new log
I20260619 06:39:52.561388 10103 tablet_bootstrap.cc:654] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:52.561528 10102 ts_tablet_manager.cc:1397] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:39:52.562266 10102 raft_consensus.cc:348] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.562517 10102 raft_consensus.cc:374] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:52.562593 10102 raft_consensus.cc:729] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2dafd092184cfda75e35989d449921, State: Initialized, Role: FOLLOWER
I20260619 06:39:52.562844 10102 consensus_queue.cc:260] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.562980 10101 tablet_bootstrap.cc:492] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1: No bootstrap required, opened a new log
I20260619 06:39:52.563115 10101 ts_tablet_manager.cc:1397] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 06:39:52.563294 10102 ts_tablet_manager.cc:1428] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:39:52.563777 10101 raft_consensus.cc:348] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.564005 10101 raft_consensus.cc:374] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:52.564093 10101 raft_consensus.cc:729] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad615111bc34050a01200f47afbfae1, State: Initialized, Role: FOLLOWER
I20260619 06:39:52.564410 10103 tablet_bootstrap.cc:492] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f: No bootstrap required, opened a new log
I20260619 06:39:52.564333 10101 consensus_queue.cc:260] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.564553 10103 ts_tablet_manager.cc:1397] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 06:39:52.564662 10068 raft_consensus.cc:482] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:39:52.564899 10101 ts_tablet_manager.cc:1428] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:39:52.564824 10068 raft_consensus.cc:504] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.565578 10068 leader_election.cc:288] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2dafd092184cfda75e35989d449921 (127.7.146.193:45073), 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799)
I20260619 06:39:52.567318 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a65279071c4211aede27de9bc109a6" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2dafd092184cfda75e35989d449921" is_pre_election: true
I20260619 06:39:52.567504 9665 raft_consensus.cc:2454] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 0.
I20260619 06:39:52.566283 9945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a65279071c4211aede27de9bc109a6" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" is_pre_election: true
I20260619 06:39:52.565435 10103 raft_consensus.cc:348] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.568068 9735 leader_election.cc:302] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad615111bc34050a01200f47afbfae1, de2dafd092184cfda75e35989d449921; no voters:
I20260619 06:39:52.568107 10103 raft_consensus.cc:374] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:52.568176 10103 raft_consensus.cc:729] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e737c9ecba4e98b1b74bc79d42a49f, State: Initialized, Role: FOLLOWER
I20260619 06:39:52.568418 10081 raft_consensus.cc:2781] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:39:52.568512 10081 raft_consensus.cc:482] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:39:52.568450 10103 consensus_queue.cc:260] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.568655 10081 raft_consensus.cc:3037] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:52.568732 9945 raft_consensus.cc:2454] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 0.
I20260619 06:39:52.569806 10103 ts_tablet_manager.cc:1428] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260619 06:39:52.571064 10081 raft_consensus.cc:504] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.571688 10081 leader_election.cc:288] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 election: Requested vote from peers de2dafd092184cfda75e35989d449921 (127.7.146.193:45073), 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799)
I20260619 06:39:52.572264 9945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a65279071c4211aede27de9bc109a6" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e737c9ecba4e98b1b74bc79d42a49f"
I20260619 06:39:52.572466 9945 raft_consensus.cc:3037] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:52.572536 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a65279071c4211aede27de9bc109a6" candidate_uuid: "5ad615111bc34050a01200f47afbfae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2dafd092184cfda75e35989d449921"
I20260619 06:39:52.572764 9665 raft_consensus.cc:3037] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:52.575289 9945 raft_consensus.cc:2454] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 1.
I20260619 06:39:52.575408 9665 raft_consensus.cc:2454] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad615111bc34050a01200f47afbfae1 in term 1.
I20260619 06:39:52.575656 9734 leader_election.cc:302] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24e737c9ecba4e98b1b74bc79d42a49f, 5ad615111bc34050a01200f47afbfae1; no voters:
I20260619 06:39:52.575914 10081 raft_consensus.cc:2781] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:52.576069 10081 raft_consensus.cc:686] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 1 LEADER]: Becoming Leader. State: Replica: 5ad615111bc34050a01200f47afbfae1, State: Running, Role: LEADER
I20260619 06:39:52.576292 10081 consensus_queue.cc:237] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } }
I20260619 06:39:52.578508 9506 catalog_manager.cc:5495] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ad615111bc34050a01200f47afbfae1 (127.7.146.194). New cstate: current_term: 1 leader_uuid: "5ad615111bc34050a01200f47afbfae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ad615111bc34050a01200f47afbfae1" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 } health_report { overall_health: UNKNOWN } } }
I20260619 06:39:52.602650 9946 raft_consensus.cc:1261] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Refusing update from remote peer 5ad615111bc34050a01200f47afbfae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:52.602882 9665 raft_consensus.cc:1261] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Refusing update from remote peer 5ad615111bc34050a01200f47afbfae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:39:52.603180 10081 consensus_queue.cc:1035] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24e737c9ecba4e98b1b74bc79d42a49f" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 36799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:39:52.603435 10068 consensus_queue.cc:1035] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2dafd092184cfda75e35989d449921" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:39:52.622135 10073 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:39:54.195817 9506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:54.196044 9506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:39:54.199944 9506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c68d2f1684748f69c02da8ed0148b84: Sending DeleteTablet for 3 replicas of tablet e9a65279071c4211aede27de9bc109a6
I20260619 06:39:54.200736 9643 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a65279071c4211aede27de9bc109a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:54 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:33224
I20260619 06:39:54.200790 9782 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a65279071c4211aede27de9bc109a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:54 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:56984
I20260619 06:39:54.201020 9921 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a65279071c4211aede27de9bc109a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:39:54 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:44558
I20260619 06:39:54.201606 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 9577
I20260619 06:39:54.213133 10142 tablet_replica.cc:330] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1: stopping tablet replica
I20260619 06:39:54.213413 10144 tablet_replica.cc:330] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f: stopping tablet replica
I20260619 06:39:54.213384 10142 raft_consensus.cc:2229] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:39:54.213595 10144 raft_consensus.cc:2229] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:54.213826 10144 raft_consensus.cc:2258] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:54.213795 10142 raft_consensus.cc:2258] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:54.214391 10143 tablet_replica.cc:330] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921: stopping tablet replica
I20260619 06:39:54.214604 10143 raft_consensus.cc:2229] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:39:54.214840 10143 raft_consensus.cc:2258] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:39:54.215430 10142 ts_tablet_manager.cc:1905] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:54.215617 10144 ts_tablet_manager.cc:1905] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:54.216048 10143 ts_tablet_manager.cc:1905] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:39:54.222055 10142 ts_tablet_manager.cc:1918] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:39:54.222203 10142 log.cc:1196] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/e9a65279071c4211aede27de9bc109a6
I20260619 06:39:54.222388 10143 ts_tablet_manager.cc:1918] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:39:54.222513 10143 log.cc:1196] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/e9a65279071c4211aede27de9bc109a6
I20260619 06:39:54.222642 10142 ts_tablet_manager.cc:1939] T e9a65279071c4211aede27de9bc109a6 P 5ad615111bc34050a01200f47afbfae1: Deleting consensus metadata
I20260619 06:39:54.222901 10143 ts_tablet_manager.cc:1939] T e9a65279071c4211aede27de9bc109a6 P de2dafd092184cfda75e35989d449921: Deleting consensus metadata
I20260619 06:39:54.223389 10144 ts_tablet_manager.cc:1918] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:39:54.223523 10144 log.cc:1196] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/e9a65279071c4211aede27de9bc109a6
I20260619 06:39:54.223773 9490 catalog_manager.cc:4841] TS 5ad615111bc34050a01200f47afbfae1 (127.7.146.194:39145): tablet e9a65279071c4211aede27de9bc109a6 (table test-table [id=735fb4f08efa424883d7932973a54637]) successfully deleted
I20260619 06:39:54.223971 10144 ts_tablet_manager.cc:1939] T e9a65279071c4211aede27de9bc109a6 P 24e737c9ecba4e98b1b74bc79d42a49f: Deleting consensus metadata
I20260619 06:39:54.224087 9492 catalog_manager.cc:4841] TS de2dafd092184cfda75e35989d449921 (127.7.146.193:45073): tablet e9a65279071c4211aede27de9bc109a6 (table test-table [id=735fb4f08efa424883d7932973a54637]) successfully deleted
I20260619 06:39:54.224969 9491 catalog_manager.cc:4841] TS 24e737c9ecba4e98b1b74bc79d42a49f (127.7.146.195:36799): tablet e9a65279071c4211aede27de9bc109a6 (table test-table [id=735fb4f08efa424883d7932973a54637]) successfully deleted
Jun 19 06:39:54 dist-test-slave-qfk4 krb5kdc[9461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:39:54.235806 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:54.202858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.7.146.193:45073)
0619 06:39:54.214672 (+ 11814us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:54.214678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:54.221176 (+ 6498us) server_negotiation.cc:407] Connection header received
0619 06:39:54.221355 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:54.221361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:54.221448 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:54.221554 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:54.224054 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:54.224971 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:54.226214 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:54.226626 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:54.226716 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:54.230507 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:54.230522 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:54.230539 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:54.230572 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:54.233251 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:54.233899 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:54.233906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:54.233909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:54.234045 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:54.234375 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:54.234379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:54.234382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:54.234723 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:54.234969 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:54.235253 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 06:39:54.235425 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11666,"thread_start_us":11965,"threads_started":1}
W20260619 06:39:54.236466 9708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55388
W20260619 06:39:54.237126 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 9577: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:54.237324 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 9577
I20260619 06:39:54.254489 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 9717
Jun 19 06:39:54 dist-test-slave-qfk4 krb5kdc[9461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:39:54.268849 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:54.255174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.7.146.194:39145)
0619 06:39:54.255647 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:54.255652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:54.255668 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:39:54.255720 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:54.255724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:54.255781 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:54.255884 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:54.257150 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:54.257868 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:54.259090 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:54.259479 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:54.259567 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:54.263718 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:54.263733 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:54.263749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:54.263785 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:54.266440 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:54.267118 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:54.267125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:54.267127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:54.267250 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:54.267559 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:54.267562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:54.267564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:54.267896 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:54.268106 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:54.268330 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 06:39:54.268512 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":242,"threads_started":1}
W20260619 06:39:54.269470 9847 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38130
W20260619 06:39:54.270043 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 9717: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:54.270187 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 9717
I20260619 06:39:54.287381 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 9856
Jun 19 06:39:54 dist-test-slave-qfk4 krb5kdc[9461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:39:54.302343 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:54.288156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.7.146.195:36799)
0619 06:39:54.288671 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:54.288675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:54.288692 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:39:54.288749 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:54.288754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:54.288820 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:54.288915 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:54.290262 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:54.291029 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:54.292249 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:54.292741 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:54.292845 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:54.296924 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:54.296941 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:54.296957 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:54.296994 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:54.299767 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:54.300402 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:54.300410 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:54.300412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:54.300599 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:54.300946 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:54.300950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:54.300953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:54.301327 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:54.301540 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:54.301818 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 06:39:54.302002 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":272,"threads_started":1}
W20260619 06:39:54.303009 9986 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43900
W20260619 06:39:54.303678 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 9856: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:39:54.303845 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 9856
I20260619 06:39:54.317099 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 9474
I20260619 06:39:54.318300 9546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:54.434712 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 9474
2026-06-19T06:39:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7733 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:54 dist-test-slave-qfk4 krb5kdc[10152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:54 dist-test-slave-qfk4 krb5kdc[10152](info): set up 2 sockets
Jun 19 06:39:54 dist-test-slave-qfk4 krb5kdc[10152](info): commencing operation
krb5kdc: starting...
W20260619 06:39:55.436959 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:39:55 dist-test-slave-qfk4 krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:39:55.618077 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:44417
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:44417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:55.928545 10165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:55.928889 10165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:55.928987 10165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:55.936174 10165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:55.936292 10165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:55.936347 10165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:55.936401 10165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:55.936451 10165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:55.946455 10165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:44417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:55.947700 10165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:55.948974 10165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:55.960281 10173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:55.960268 10171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:55.960325 10170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:55.961009 10165 server_base.cc:1034] running on GCE node
I20260619 06:39:55.961751 10165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:55.963141 10165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:55.964349 10165 hybrid_clock.cc:648] HybridClock initialized: now 1781851195964291 us; error 48 us; skew 500 ppm
I20260619 06:39:55.964713 10165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:55 dist-test-slave-qfk4 krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851195, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:55.970745 10165 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:39:55.972271 10165 webserver.cc:460] Webserver started at http://127.7.146.254:44577/ using document root <none> and password file <none>
I20260619 06:39:55.972986 10165 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:55.973102 10165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:55.973377 10165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:55.976269 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "024083ee746c4b39b6acebcba56ab86f"
format_stamp: "Formatted at 2026-06-19 06:39:55 on dist-test-slave-qfk4"
server_key: "7cdffb4d6e962fd790d191652cc485a6"
server_key_iv: "afdcc4fe8d0f0451b6fd56f4ed8c0034"
server_key_version: "encryptionkey@0"
I20260619 06:39:55.977023 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "024083ee746c4b39b6acebcba56ab86f"
format_stamp: "Formatted at 2026-06-19 06:39:55 on dist-test-slave-qfk4"
server_key: "7cdffb4d6e962fd790d191652cc485a6"
server_key_iv: "afdcc4fe8d0f0451b6fd56f4ed8c0034"
server_key_version: "encryptionkey@0"
I20260619 06:39:55.981890 10165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 06:39:55.985144 10180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:55.986442 10165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:55.986573 10165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "024083ee746c4b39b6acebcba56ab86f"
format_stamp: "Formatted at 2026-06-19 06:39:55 on dist-test-slave-qfk4"
server_key: "7cdffb4d6e962fd790d191652cc485a6"
server_key_iv: "afdcc4fe8d0f0451b6fd56f4ed8c0034"
server_key_version: "encryptionkey@0"
I20260619 06:39:55.986733 10165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:56.005869 10165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:56.010385 10165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:56.010614 10165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:56.027786 10165 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:44417
I20260619 06:39:56.027815 10231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:44417 every 8 connection(s)
I20260619 06:39:56.029210 10165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:56.032869 10232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:39:56.034626 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 10165
I20260619 06:39:56.034847 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:39:56.035583 7755 external_mini_cluster.cc:1428] Setting key 56f5d16744bc05fdbafbbb4f06eeaf8c
I20260619 06:39:56.044998 10232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f: Bootstrap starting.
I20260619 06:39:56.048645 10232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:56.049844 10232 log.cc:824] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:39:56 dist-test-slave-qfk4 krb5kdc[10152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:56.052536 10232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f: No bootstrap required, opened a new log
I20260619 06:39:56.058311 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:56.037652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.7.146.254:44417)
0619 06:39:56.038435 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:56.038451 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:56.038490 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:39:56.040156 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:56.040179 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:56.040646 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:56.040980 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:56.042660 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:56.044089 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:56.045845 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:56.046535 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:56.046670 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:56.051107 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:56.051115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:56.051130 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:56.051164 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:56.053994 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:56.054790 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:56.054799 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:56.054808 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:56.055022 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:56.055449 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:56.055454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:56.055456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:56.056176 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:56.056413 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:56.056864 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 06:39:56.057240 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":348,"threads_started":1}
I20260619 06:39:56.060907 10232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024083ee746c4b39b6acebcba56ab86f" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44417 } }
I20260619 06:39:56.061237 10232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:56.061334 10232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 024083ee746c4b39b6acebcba56ab86f, State: Initialized, Role: FOLLOWER
I20260619 06:39:56.061843 10232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024083ee746c4b39b6acebcba56ab86f" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44417 } }
I20260619 06:39:56.062065 10232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:56.062178 10232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:56.062317 10232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:56.064378 10232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024083ee746c4b39b6acebcba56ab86f" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44417 } }
I20260619 06:39:56.064879 10232 leader_election.cc:302] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 024083ee746c4b39b6acebcba56ab86f; no voters:
I20260619 06:39:56.065486 10232 leader_election.cc:288] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:56.065613 10237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:56.066239 10237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [term 1 LEADER]: Becoming Leader. State: Replica: 024083ee746c4b39b6acebcba56ab86f, State: Running, Role: LEADER
I20260619 06:39:56.066849 10237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024083ee746c4b39b6acebcba56ab86f" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44417 } }
I20260619 06:39:56.067498 10232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:56.069931 10238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "024083ee746c4b39b6acebcba56ab86f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024083ee746c4b39b6acebcba56ab86f" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44417 } } }
I20260619 06:39:56.069948 10239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024083ee746c4b39b6acebcba56ab86f. Latest consensus state: current_term: 1 leader_uuid: "024083ee746c4b39b6acebcba56ab86f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "024083ee746c4b39b6acebcba56ab86f" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 44417 } } }
I20260619 06:39:56.070348 10238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:56.070349 10239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:56.072297 10244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:56.077646 10244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:56.086876 10244 catalog_manager.cc:1347] Generated new cluster ID: 666a0211ff774d5a8ca986a1cf7f1f7f
I20260619 06:39:56.087018 10244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:56.105509 10244 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:56.106849 10244 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:56.112691 10244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 024083ee746c4b39b6acebcba56ab86f: Generated new TSK 0
I20260619 06:39:56.113521 10244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:39:56.159179 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44417
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:56.520854 10258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:56.521155 10258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:56.521258 10258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:56.528532 10258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:56.528705 10258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:56.528841 10258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:39:56.538637 10258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:56.539913 10258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:56.541363 10258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:56.554543 10266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:56.554558 10263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:56.554626 10264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:56.555239 10258 server_base.cc:1034] running on GCE node
I20260619 06:39:56.555868 10258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:56.556870 10258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:56.558105 10258 hybrid_clock.cc:648] HybridClock initialized: now 1781851196558048 us; error 46 us; skew 500 ppm
I20260619 06:39:56.558485 10258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:56 dist-test-slave-qfk4 krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851196, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:56.561996 10258 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:39:56.563608 10258 webserver.cc:460] Webserver started at http://127.7.146.193:32849/ using document root <none> and password file <none>
I20260619 06:39:56.564311 10258 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:56.564426 10258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:56.564764 10258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:56.567673 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "a68e5ce870d84f5d9ee4c2b874c9daab"
format_stamp: "Formatted at 2026-06-19 06:39:56 on dist-test-slave-qfk4"
server_key: "c9b11e5afc3e00b1f702a56d8529acf1"
server_key_iv: "0eb14098610a305cd8b774ac055fc634"
server_key_version: "encryptionkey@0"
I20260619 06:39:56.568522 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "a68e5ce870d84f5d9ee4c2b874c9daab"
format_stamp: "Formatted at 2026-06-19 06:39:56 on dist-test-slave-qfk4"
server_key: "c9b11e5afc3e00b1f702a56d8529acf1"
server_key_iv: "0eb14098610a305cd8b774ac055fc634"
server_key_version: "encryptionkey@0"
I20260619 06:39:56.573930 10258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:39:56.577402 10273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:56.578882 10258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:39:56.579046 10258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "a68e5ce870d84f5d9ee4c2b874c9daab"
format_stamp: "Formatted at 2026-06-19 06:39:56 on dist-test-slave-qfk4"
server_key: "c9b11e5afc3e00b1f702a56d8529acf1"
server_key_iv: "0eb14098610a305cd8b774ac055fc634"
server_key_version: "encryptionkey@0"
I20260619 06:39:56.579228 10258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:56.593766 10258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:56.597723 10258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:56.597994 10258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:56.598934 10258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:39:56.600241 10258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:56.600311 10258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:56.600426 10258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:56.600502 10258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:56.630168 10258 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:35567
I20260619 06:39:56.630214 10385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:35567 every 8 connection(s)
I20260619 06:39:56.631412 10258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:39:56.641673 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 10258
I20260619 06:39:56.641891 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:39:56.642670 7755 external_mini_cluster.cc:1428] Setting key e39b3470d6142a9bdd288f47af0386db
Jun 19 06:39:56 dist-test-slave-qfk4 krb5kdc[10152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851196, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:56.650214 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:56.633722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:52791 (local address 127.7.146.254:44417)
0619 06:39:56.634231 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:56.634237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:56.635126 (+ 889us) server_negotiation.cc:407] Connection header received
0619 06:39:56.636898 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:56.636904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:56.636968 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:56.637050 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:56.638483 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:56.639162 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:56.640664 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:56.641042 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:56.641132 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:56.645121 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:56.645126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:56.645128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:56.645158 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:56.647436 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:56.648108 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:56.648112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:56.648114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:56.648232 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:56.648637 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:56.648641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:56.648643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:56.648886 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:56.649014 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:56.649679 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 06:39:56.649888 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":199,"threads_started":1}
I20260619 06:39:56.651166 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:56.634209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:44417 (local address 127.7.146.193:52791)
0619 06:39:56.634955 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:56.635001 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:56.636632 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:56.637242 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:56.637250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:56.637726 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:56.638349 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:56.638367 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:56.639282 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:56.639285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:56.640509 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:56.640520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:56.641255 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:56.641262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:56.641570 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:56.642268 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:56.642298 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:56.644921 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:56.647560 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:56.647569 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:56.647586 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:56.647991 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:56.648329 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:56.648334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:56.648338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:56.648517 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:56.649002 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:56.649010 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:56.649328 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 06:39:56.649553 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 06:39:56.650071 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":264,"threads_started":1}
I20260619 06:39:56.652645 10386 heartbeater.cc:347] Connected to a master server at 127.7.146.254:44417
I20260619 06:39:56.652909 10386 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:56.653591 10386 heartbeater.cc:511] Master 127.7.146.254:44417 requested a full tablet report, sending...
I20260619 06:39:56.655802 10197 ts_manager.cc:194] Registered new tserver with Master: a68e5ce870d84f5d9ee4c2b874c9daab (127.7.146.193:35567)
I20260619 06:39:56.657886 10197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:52791
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:39:56.681509 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44417
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:56.996395 10392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:56.996708 10392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:56.996850 10392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:57.004097 10392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:57.004228 10392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:57.004388 10392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:39:57.014093 10392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:57.015431 10392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:57.016749 10392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:57.029052 10397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:57.029053 10398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:57.029814 10400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:57.030368 10392 server_base.cc:1034] running on GCE node
I20260619 06:39:57.030946 10392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:57.031844 10392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:57.033046 10392 hybrid_clock.cc:648] HybridClock initialized: now 1781851197032995 us; error 45 us; skew 500 ppm
I20260619 06:39:57.033349 10392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:57 dist-test-slave-qfk4 krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851197, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:57.036350 10392 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:39:57.037844 10392 webserver.cc:460] Webserver started at http://127.7.146.194:45083/ using document root <none> and password file <none>
I20260619 06:39:57.038489 10392 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:57.038589 10392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:57.038883 10392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:57.041785 10392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "9cccaec8e21245cca059e19e1ee0d170"
format_stamp: "Formatted at 2026-06-19 06:39:57 on dist-test-slave-qfk4"
server_key: "e135f86752d22e377c650348aead4ab5"
server_key_iv: "591a9fd1dadc606407a5565d3ac83b90"
server_key_version: "encryptionkey@0"
I20260619 06:39:57.042518 10392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "9cccaec8e21245cca059e19e1ee0d170"
format_stamp: "Formatted at 2026-06-19 06:39:57 on dist-test-slave-qfk4"
server_key: "e135f86752d22e377c650348aead4ab5"
server_key_iv: "591a9fd1dadc606407a5565d3ac83b90"
server_key_version: "encryptionkey@0"
I20260619 06:39:57.047621 10392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:39:57.050935 10407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:57.052315 10392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:39:57.052476 10392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "9cccaec8e21245cca059e19e1ee0d170"
format_stamp: "Formatted at 2026-06-19 06:39:57 on dist-test-slave-qfk4"
server_key: "e135f86752d22e377c650348aead4ab5"
server_key_iv: "591a9fd1dadc606407a5565d3ac83b90"
server_key_version: "encryptionkey@0"
I20260619 06:39:57.052713 10392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:57.083813 10392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:57.088304 10392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:57.088616 10392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:57.089609 10392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:39:57.091012 10392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:57.091092 10392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:57.091210 10392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:57.091259 10392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:57.123098 10392 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:43099
I20260619 06:39:57.123152 10519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:43099 every 8 connection(s)
I20260619 06:39:57.124686 10392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:39:57.128544 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 10392
I20260619 06:39:57.128790 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:39:57.129309 7755 external_mini_cluster.cc:1428] Setting key cb1fd24d78f8041d564f29628487609f
Jun 19 06:39:57 dist-test-slave-qfk4 krb5kdc[10152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851197, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
I20260619 06:39:57.145787 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:57.127316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:37089 (local address 127.7.146.254:44417)
0619 06:39:57.127481 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:57.127485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:57.128907 (+ 1422us) server_negotiation.cc:407] Connection header received
0619 06:39:57.130901 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:57.130907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:57.130996 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:57.131091 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:57.132771 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:57.133495 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:57.135165 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:57.135584 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:57.135685 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:57.139558 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:57.139563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:57.139566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:57.139600 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:57.142546 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:57.143337 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:57.143343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:57.143347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:57.143501 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:57.143920 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:57.143925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:57.143928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:57.144281 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:57.144424 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:57.145207 (+ 783us) server_negotiation.cc:299] Negotiation successful
0619 06:39:57.145471 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:39:57.146842 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:57.127884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:44417 (local address 127.7.146.194:37089)
0619 06:39:57.128701 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:57.128776 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:57.130607 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:57.131357 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:57.131367 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:57.131905 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:57.132512 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:57.132536 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:57.133722 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:57.133729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:57.135008 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:57.135021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:57.135800 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:57.135808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:57.136117 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:57.136839 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:57.136868 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:57.139326 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:57.142711 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:57.142718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:57.142733 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:57.143199 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:57.143616 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:57.143619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:57.143621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:57.143802 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:57.144404 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:57.144410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:57.144782 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 06:39:57.145033 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 06:39:57.145570 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":337,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
I20260619 06:39:57.148437 10520 heartbeater.cc:347] Connected to a master server at 127.7.146.254:44417
I20260619 06:39:57.148753 10520 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:57.149466 10520 heartbeater.cc:511] Master 127.7.146.254:44417 requested a full tablet report, sending...
I20260619 06:39:57.150974 10197 ts_manager.cc:194] Registered new tserver with Master: 9cccaec8e21245cca059e19e1ee0d170 (127.7.146.194:43099)
I20260619 06:39:57.151783 10197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:37089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:39:57.169657 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44417
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:57.482208 10525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:57.482481 10525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:57.482599 10525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:57.489816 10525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:57.489923 10525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:57.490064 10525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:39:57.499578 10525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:44417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:57.500989 10525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:57.502424 10525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:57.514611 10531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:57.514722 10530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:57.514586 10533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:57.518186 10525 server_base.cc:1034] running on GCE node
I20260619 06:39:57.518785 10525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:57.519745 10525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:57.520953 10525 hybrid_clock.cc:648] HybridClock initialized: now 1781851197520903 us; error 42 us; skew 500 ppm
I20260619 06:39:57.521252 10525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:57 dist-test-slave-qfk4 krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851197, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:57.524279 10525 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:39:57.525915 10525 webserver.cc:460] Webserver started at http://127.7.146.195:39577/ using document root <none> and password file <none>
I20260619 06:39:57.526635 10525 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:57.526737 10525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:57.527005 10525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:57.529867 10525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "54726f87202b4216940d517e19831568"
format_stamp: "Formatted at 2026-06-19 06:39:57 on dist-test-slave-qfk4"
server_key: "564ae665b8b500a2b57391e482e1403f"
server_key_iv: "f83202d89c76cf6cc69edec71ef68437"
server_key_version: "encryptionkey@0"
I20260619 06:39:57.530591 10525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "54726f87202b4216940d517e19831568"
format_stamp: "Formatted at 2026-06-19 06:39:57 on dist-test-slave-qfk4"
server_key: "564ae665b8b500a2b57391e482e1403f"
server_key_iv: "f83202d89c76cf6cc69edec71ef68437"
server_key_version: "encryptionkey@0"
I20260619 06:39:57.535740 10525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:39:57.539237 10540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:57.540810 10525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:39:57.540987 10525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "54726f87202b4216940d517e19831568"
format_stamp: "Formatted at 2026-06-19 06:39:57 on dist-test-slave-qfk4"
server_key: "564ae665b8b500a2b57391e482e1403f"
server_key_iv: "f83202d89c76cf6cc69edec71ef68437"
server_key_version: "encryptionkey@0"
I20260619 06:39:57.541208 10525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:57.567247 10525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:57.571269 10525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:57.571525 10525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:57.572541 10525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:39:57.573923 10525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:39:57.573992 10525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:57.574095 10525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:39:57.574141 10525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:57.603819 10525 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:44425
I20260619 06:39:57.603956 10652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:44425 every 8 connection(s)
I20260619 06:39:57.605063 10525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:39:57.606411 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 10525
I20260619 06:39:57.606594 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:39:57.606979 7755 external_mini_cluster.cc:1428] Setting key 7c60cc4f929f2a889f59bbcea8cb6a15
Jun 19 06:39:57 dist-test-slave-qfk4 krb5kdc[10152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851197, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:57.624127 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:57.607735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:48231 (local address 127.7.146.254:44417)
0619 06:39:57.607941 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:57.607948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:57.609273 (+ 1325us) server_negotiation.cc:407] Connection header received
0619 06:39:57.611347 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:57.611353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:57.611439 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:57.611527 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:57.613011 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:57.613728 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:57.615226 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:57.615616 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:57.615720 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:57.618941 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:57.618945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:57.618949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:57.618980 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:57.621270 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:57.621948 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:57.621952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:57.621954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:57.622081 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:57.622472 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:57.622476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:57.622478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:57.622730 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:57.622852 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:57.623537 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 06:39:57.623743 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:39:57.625139 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:57.608316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:44417 (local address 127.7.146.195:48231)
0619 06:39:57.609129 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 06:39:57.609169 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:39:57.611072 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:39:57.611773 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:39:57.611783 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:39:57.612261 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:39:57.612862 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:57.612880 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:57.613861 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:57.613865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:57.615097 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:39:57.615109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:39:57.615817 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:39:57.615824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:39:57.616046 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:57.616670 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:39:57.616692 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:39:57.618745 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:39:57.621399 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:57.621408 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:57.621425 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:57.621827 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:57.622179 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:39:57.622184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:39:57.622188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:39:57.622360 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:39:57.622851 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:39:57.622859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:39:57.623185 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 06:39:57.623421 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 06:39:57.623925 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":327,"threads_started":1}
I20260619 06:39:57.626402 10653 heartbeater.cc:347] Connected to a master server at 127.7.146.254:44417
I20260619 06:39:57.626693 10653 heartbeater.cc:464] Registering TS with master...
I20260619 06:39:57.627385 10653 heartbeater.cc:511] Master 127.7.146.254:44417 requested a full tablet report, sending...
I20260619 06:39:57.628772 10197 ts_manager.cc:194] Registered new tserver with Master: 54726f87202b4216940d517e19831568 (127.7.146.195:44425)
I20260619 06:39:57.629567 10197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:48231
I20260619 06:39:57.634581 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:39:57.661028 10388 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:39:57.658576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.7.146.254:44417)
0619 06:39:57.658757 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:57.658762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:57.659227 (+ 465us) server_negotiation.cc:407] Connection header received
0619 06:39:57.659368 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:57.659372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:57.659436 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:57.659529 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:57.660699 (+ 1170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59204: BlockingRecv error: recv got EOF from 127.0.0.1:59204 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:39:57.661126 10386 heartbeater.cc:503] Master 127.7.146.254:44417 was elected leader, sending a full tablet report...
I20260619 06:39:57.664472 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 10258
W20260619 06:39:57.667204 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 10258: Not authorized: Client connection negotiation failed: client connection to 127.7.146.193:35567: server requires authentication, but client does not have Kerberos credentials available
I20260619 06:39:57.667382 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 10258
I20260619 06:39:57.679317 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 10392
W20260619 06:39:57.682037 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 10392: Not authorized: Client connection negotiation failed: client connection to 127.7.146.194:43099: server requires authentication, but client does not have Kerberos credentials available
I20260619 06:39:57.682195 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 10392
I20260619 06:39:57.693711 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 10525
W20260619 06:39:57.696653 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 10525: Not authorized: Client connection negotiation failed: client connection to 127.7.146.195:44425: server requires authentication, but client does not have Kerberos credentials available
I20260619 06:39:57.696813 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 10525
I20260619 06:39:57.708191 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 10165
I20260619 06:39:57.709455 10227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:57.810250 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 10165
2026-06-19T06:39:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3373 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:57 dist-test-slave-qfk4 krb5kdc[10670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:57 dist-test-slave-qfk4 krb5kdc[10670](info): set up 2 sockets
Jun 19 06:39:57 dist-test-slave-qfk4 krb5kdc[10670](info): commencing operation
krb5kdc: starting...
W20260619 06:39:58.812608 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:39:58 dist-test-slave-qfk4 krb5kdc[10670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:39:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:39:58.991707 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:43255
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:43255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:39:59.301263 10683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:39:59.301548 10683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:39:59.301632 10683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:39:59.308861 10683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:39:59.308966 10683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:39:59.309010 10683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:39:59.309063 10683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:39:59.309108 10683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:39:59.319267 10683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:43255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:43255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:39:59.320672 10683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:39:59.322011 10683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:39:59.333667 10688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:59.333633 10689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:39:59.333691 10691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:39:59.334229 10683 server_base.cc:1034] running on GCE node
I20260619 06:39:59.335005 10683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:39:59.336489 10683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:39:59.337769 10683 hybrid_clock.cc:648] HybridClock initialized: now 1781851199337723 us; error 34 us; skew 500 ppm
I20260619 06:39:59.338124 10683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:39:59 dist-test-slave-qfk4 krb5kdc[10670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851199, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:39:59.341234 10683 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:39:59.342630 10683 webserver.cc:460] Webserver started at http://127.7.146.254:39995/ using document root <none> and password file <none>
I20260619 06:39:59.343286 10683 fs_manager.cc:359] Metadata directory not provided
I20260619 06:39:59.343398 10683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:39:59.343674 10683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:39:59.346534 10683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "01a682fac19e45be9735bc4e9dae29a8"
format_stamp: "Formatted at 2026-06-19 06:39:59 on dist-test-slave-qfk4"
server_key: "f6d5eae0e1db0d5e3057ee04aa057e7c"
server_key_iv: "07d8cb915c21cc1e2cea011ac7340bc8"
server_key_version: "encryptionkey@0"
I20260619 06:39:59.347278 10683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "01a682fac19e45be9735bc4e9dae29a8"
format_stamp: "Formatted at 2026-06-19 06:39:59 on dist-test-slave-qfk4"
server_key: "f6d5eae0e1db0d5e3057ee04aa057e7c"
server_key_iv: "07d8cb915c21cc1e2cea011ac7340bc8"
server_key_version: "encryptionkey@0"
I20260619 06:39:59.352190 10683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:39:59.355674 10698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:39:59.357059 10683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:39:59.357211 10683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "01a682fac19e45be9735bc4e9dae29a8"
format_stamp: "Formatted at 2026-06-19 06:39:59 on dist-test-slave-qfk4"
server_key: "f6d5eae0e1db0d5e3057ee04aa057e7c"
server_key_iv: "07d8cb915c21cc1e2cea011ac7340bc8"
server_key_version: "encryptionkey@0"
I20260619 06:39:59.357412 10683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:39:59.372781 10683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:39:59.376961 10683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:39:59.377205 10683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:39:59.395581 10683 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:43255
I20260619 06:39:59.395625 10749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:43255 every 8 connection(s)
I20260619 06:39:59.396958 10683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:39:59.398169 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 10683
I20260619 06:39:59.398360 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:39:59.398952 7755 external_mini_cluster.cc:1428] Setting key dcffc0cacbf127741a7dc42e802f5456
I20260619 06:39:59.401124 10750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:39:59 dist-test-slave-qfk4 krb5kdc[10670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:39:59.413558 10750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8: Bootstrap starting.
I20260619 06:39:59.417608 10750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8: Neither blocks nor log segments found. Creating new log.
I20260619 06:39:59.418766 10750 log.cc:824] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:39:59.422312 10750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8: No bootstrap required, opened a new log
I20260619 06:39:59.422755 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:59.400759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.7.146.254:43255)
0619 06:39:59.401497 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:59.401508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:59.401546 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:39:59.403158 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:59.403179 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:59.403573 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:59.403930 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:59.405674 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:59.407040 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:59.408701 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:59.409356 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:59.409492 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:59.413820 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:59.413831 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:59.413850 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:39:59.413897 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:59.417550 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:59.418476 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:59.418485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:59.418494 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:59.418693 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:39:59.419115 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:39:59.419138 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:39:59.419141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:39:59.419948 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:59.420243 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:59.420872 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 06:39:59.421312 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":331,"threads_started":1}
I20260619 06:39:59.427214 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:39:59.432224 10750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a682fac19e45be9735bc4e9dae29a8" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43255 } }
I20260619 06:39:59.432667 10750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:39:59.432765 10750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a682fac19e45be9735bc4e9dae29a8, State: Initialized, Role: FOLLOWER
I20260619 06:39:59.433377 10750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a682fac19e45be9735bc4e9dae29a8" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43255 } }
I20260619 06:39:59.433627 10750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:39:59.433748 10750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:39:59.433910 10750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:39:59.436101 10750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a682fac19e45be9735bc4e9dae29a8" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43255 } }
I20260619 06:39:59.436580 10750 leader_election.cc:302] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01a682fac19e45be9735bc4e9dae29a8; no voters:
I20260619 06:39:59.437139 10750 leader_election.cc:288] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:39:59.437291 10756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:39:59.439050 10750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:39:59.440831 10756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [term 1 LEADER]: Becoming Leader. State: Replica: 01a682fac19e45be9735bc4e9dae29a8, State: Running, Role: LEADER
I20260619 06:39:59.441460 10756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a682fac19e45be9735bc4e9dae29a8" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43255 } }
I20260619 06:39:59.444059 10758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01a682fac19e45be9735bc4e9dae29a8. Latest consensus state: current_term: 1 leader_uuid: "01a682fac19e45be9735bc4e9dae29a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a682fac19e45be9735bc4e9dae29a8" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43255 } } }
I20260619 06:39:59.444429 10758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:59.444967 10757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01a682fac19e45be9735bc4e9dae29a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a682fac19e45be9735bc4e9dae29a8" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43255 } } }
I20260619 06:39:59.445313 10757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:39:59.449733 10764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:39:59.455164 10764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:39:59.465107 10775 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:39:59.471306 10764 catalog_manager.cc:1347] Generated new cluster ID: f8703ac3c4af466891efe16b113e7bc1
I20260619 06:39:59.471419 10764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:39:59.483412 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:39:59.476837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.7.146.254:43255)
0619 06:39:59.477011 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:39:59.477015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:39:59.477032 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:39:59.477083 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:39:59.477087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:39:59.477151 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:39:59.477253 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:39:59.478277 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:59.479059 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:59.480308 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:39:59.480829 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:39:59.480964 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:39:59.481974 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:39:59.481979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:39:59.481982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:39:59.482648 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:39:59.482692 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:39:59.482721 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:39:59.482811 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:39:59.482961 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 06:39:59.483056 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:39:59.498626 10764 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:39:59.500537 10764 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:39:59.516247 10764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01a682fac19e45be9735bc4e9dae29a8: Generated new TSK 0
I20260619 06:39:59.517134 10764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:39:59.526881 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 10683
I20260619 06:39:59.529510 10745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:39:59.619378 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 10683
2026-06-19T06:39:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1803 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:39:59 dist-test-slave-qfk4 krb5kdc[10785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:39:59 dist-test-slave-qfk4 krb5kdc[10785](info): set up 2 sockets
Jun 19 06:39:59 dist-test-slave-qfk4 krb5kdc[10785](info): commencing operation
krb5kdc: starting...
W20260619 06:40:00.604730 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:40:00 dist-test-slave-qfk4 krb5kdc[10785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:40:00.782213 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:43955
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:43955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:01.091797 10798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:01.092110 10798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:01.092254 10798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:01.099550 10798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:01.099680 10798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:01.099766 10798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:01.099846 10798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:01.099905 10798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:01.110963 10798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:43955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:43955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:01.112355 10798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:01.113708 10798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:01.125358 10806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:01.125332 10804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:01.125396 10803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:01.130196 10798 server_base.cc:1034] running on GCE node
I20260619 06:40:01.130962 10798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:01.132395 10798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:01.133594 10798 hybrid_clock.cc:648] HybridClock initialized: now 1781851201133528 us; error 56 us; skew 500 ppm
I20260619 06:40:01.133910 10798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:01 dist-test-slave-qfk4 krb5kdc[10785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851201, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:01.139824 10798 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:40:01.141407 10798 webserver.cc:460] Webserver started at http://127.7.146.254:44971/ using document root <none> and password file <none>
I20260619 06:40:01.142107 10798 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:01.142203 10798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:01.142473 10798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:01.145488 10798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "3284b27bf8e94d8e8bd40ff380e7d5a1"
format_stamp: "Formatted at 2026-06-19 06:40:01 on dist-test-slave-qfk4"
server_key: "c8965a5bdb4e5008d309d98101db7dbf"
server_key_iv: "8111b06b3558aebc3f7591d8aabe4b13"
server_key_version: "encryptionkey@0"
I20260619 06:40:01.146163 10798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "3284b27bf8e94d8e8bd40ff380e7d5a1"
format_stamp: "Formatted at 2026-06-19 06:40:01 on dist-test-slave-qfk4"
server_key: "c8965a5bdb4e5008d309d98101db7dbf"
server_key_iv: "8111b06b3558aebc3f7591d8aabe4b13"
server_key_version: "encryptionkey@0"
I20260619 06:40:01.151438 10798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:40:01.154942 10813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:01.156257 10798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 06:40:01.156426 10798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "3284b27bf8e94d8e8bd40ff380e7d5a1"
format_stamp: "Formatted at 2026-06-19 06:40:01 on dist-test-slave-qfk4"
server_key: "c8965a5bdb4e5008d309d98101db7dbf"
server_key_iv: "8111b06b3558aebc3f7591d8aabe4b13"
server_key_version: "encryptionkey@0"
I20260619 06:40:01.156646 10798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:01.175904 10798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:01.180153 10798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:01.180424 10798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:01.197587 10798 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:43955
I20260619 06:40:01.197633 10864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:43955 every 8 connection(s)
I20260619 06:40:01.198853 10798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:01.202601 10865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:01.208904 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 10798
I20260619 06:40:01.209110 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:01.209913 7755 external_mini_cluster.cc:1428] Setting key e2bc7071f1647a22f923f3ab2bf15795
I20260619 06:40:01.213418 10865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1: Bootstrap starting.
I20260619 06:40:01.216852 10865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:01.217845 10865 log.cc:824] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:01.220775 10865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1: No bootstrap required, opened a new log
Jun 19 06:40:01 dist-test-slave-qfk4 krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:01.229719 10865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b27bf8e94d8e8bd40ff380e7d5a1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43955 } }
I20260619 06:40:01.230134 10865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:01.230237 10865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3284b27bf8e94d8e8bd40ff380e7d5a1, State: Initialized, Role: FOLLOWER
I20260619 06:40:01.230897 10865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b27bf8e94d8e8bd40ff380e7d5a1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43955 } }
I20260619 06:40:01.231148 10865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:01.231292 10865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:01.231487 10865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:01.232796 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:01.211963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.7.146.254:43955)
0619 06:40:01.212702 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:01.212714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:01.212754 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:40:01.214403 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:01.214426 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:01.214817 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:01.215158 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:01.216810 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:01.218146 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:01.219609 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:01.220261 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:01.220390 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:01.225299 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:01.225307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:01.225326 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:01.225366 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:01.228469 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:01.229145 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:01.229151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:01.229157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:01.229333 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:01.229713 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:01.229716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:01.229718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:01.230339 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:01.230656 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:01.231061 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 06:40:01.231626 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":333,"threads_started":1}
I20260619 06:40:01.234055 10865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b27bf8e94d8e8bd40ff380e7d5a1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43955 } }
I20260619 06:40:01.234582 10865 leader_election.cc:302] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3284b27bf8e94d8e8bd40ff380e7d5a1; no voters:
I20260619 06:40:01.235237 10865 leader_election.cc:288] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:01.235375 10870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:01.236064 10870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [term 1 LEADER]: Becoming Leader. State: Replica: 3284b27bf8e94d8e8bd40ff380e7d5a1, State: Running, Role: LEADER
I20260619 06:40:01.236670 10870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b27bf8e94d8e8bd40ff380e7d5a1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43955 } }
I20260619 06:40:01.237674 10865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:01.239507 10872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3284b27bf8e94d8e8bd40ff380e7d5a1. Latest consensus state: current_term: 1 leader_uuid: "3284b27bf8e94d8e8bd40ff380e7d5a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b27bf8e94d8e8bd40ff380e7d5a1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43955 } } }
I20260619 06:40:01.239825 10872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:01.240582 10871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3284b27bf8e94d8e8bd40ff380e7d5a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3284b27bf8e94d8e8bd40ff380e7d5a1" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43955 } } }
I20260619 06:40:01.240897 10871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:01.245801 10876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:01.252038 10876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:01.261854 10876 catalog_manager.cc:1347] Generated new cluster ID: 5f28780f7a4a4427bc56b5e1142411e8
I20260619 06:40:01.261970 10876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:01.271375 10876 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:01.273291 10876 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:01.284394 10876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3284b27bf8e94d8e8bd40ff380e7d5a1: Generated new TSK 0
I20260619 06:40:01.285316 10876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:40:01.334417 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43955
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:01.652891 10891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:01.653189 10891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:01.653316 10891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:01.660730 10891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:01.660866 10891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:01.661039 10891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:40:01.670737 10891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:01.672046 10891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:01.673365 10891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:01.686022 10899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:01.686059 10897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:01.685989 10896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:01.686817 10891 server_base.cc:1034] running on GCE node
I20260619 06:40:01.687454 10891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:01.688392 10891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:01.689610 10891 hybrid_clock.cc:648] HybridClock initialized: now 1781851201689564 us; error 61 us; skew 500 ppm
I20260619 06:40:01.689918 10891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:01 dist-test-slave-qfk4 krb5kdc[10785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851201, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:01.693239 10891 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:40:01.694674 10891 webserver.cc:460] Webserver started at http://127.7.146.193:40173/ using document root <none> and password file <none>
I20260619 06:40:01.695326 10891 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:01.695423 10891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:01.695672 10891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:01.698635 10891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "509739b6da1343d19d9b7489502b8093"
format_stamp: "Formatted at 2026-06-19 06:40:01 on dist-test-slave-qfk4"
server_key: "373f2e290e5b3c74a4867083f146e35b"
server_key_iv: "86d7ef3cfe6b4103bdee8106ec081ac4"
server_key_version: "encryptionkey@0"
I20260619 06:40:01.699342 10891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "509739b6da1343d19d9b7489502b8093"
format_stamp: "Formatted at 2026-06-19 06:40:01 on dist-test-slave-qfk4"
server_key: "373f2e290e5b3c74a4867083f146e35b"
server_key_iv: "86d7ef3cfe6b4103bdee8106ec081ac4"
server_key_version: "encryptionkey@0"
I20260619 06:40:01.704514 10891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:40:01.708352 10906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:01.709856 10891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:01.709995 10891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "509739b6da1343d19d9b7489502b8093"
format_stamp: "Formatted at 2026-06-19 06:40:01 on dist-test-slave-qfk4"
server_key: "373f2e290e5b3c74a4867083f146e35b"
server_key_iv: "86d7ef3cfe6b4103bdee8106ec081ac4"
server_key_version: "encryptionkey@0"
I20260619 06:40:01.710155 10891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:01.731464 10891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:01.735775 10891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:01.736040 10891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:01.737042 10891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:01.738416 10891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:01.738538 10891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:01.738660 10891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:01.738718 10891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:01.769038 10891 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:35411
I20260619 06:40:01.769096 11018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:35411 every 8 connection(s)
I20260619 06:40:01.770325 10891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:40:01.773243 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 10891
I20260619 06:40:01.773471 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:40:01.774128 7755 external_mini_cluster.cc:1428] Setting key 1d1504032471165e8eac5aa9db6cc971
Jun 19 06:40:01 dist-test-slave-qfk4 krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851201, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:01.790858 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:01.773110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:36143 (local address 127.7.146.254:43955)
0619 06:40:01.773579 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:01.773585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:01.776884 (+ 3299us) server_negotiation.cc:407] Connection header received
0619 06:40:01.776947 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:01.776951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:01.777016 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:01.777111 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:01.778697 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:01.779429 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:01.781039 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:01.781430 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:01.781543 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:01.785170 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:01.785175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:01.785178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:01.785210 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:01.787574 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:01.788518 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:01.788522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:01.788525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:01.788675 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:01.789144 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:01.789147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:01.789149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:01.789403 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:01.789510 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:01.790403 (+ 893us) server_negotiation.cc:299] Negotiation successful
0619 06:40:01.790596 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
I20260619 06:40:01.792418 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:01.773683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:43955 (local address 127.7.146.193:36143)
0619 06:40:01.774458 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:01.774523 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:01.776420 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:01.777390 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:01.777399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:01.777934 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:01.778524 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:01.778543 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:01.779578 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:01.779582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:01.780902 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:01.780914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:01.781645 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:01.781652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:01.781882 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:01.782515 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:01.782538 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:01.784897 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:01.787733 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:01.787742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:01.787760 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:01.788376 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:01.788776 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:01.788781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:01.788785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:01.789045 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:01.789523 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:01.789532 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:01.790024 (+ 492us) client_negotiation.cc:769] Sending connection context
0619 06:40:01.790339 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 06:40:01.790950 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":305,"threads_started":1}
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
I20260619 06:40:01.794035 11019 heartbeater.cc:347] Connected to a master server at 127.7.146.254:43955
I20260619 06:40:01.794325 11019 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:01.795050 11019 heartbeater.cc:511] Master 127.7.146.254:43955 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
I20260619 06:40:01.797252 10830 ts_manager.cc:194] Registered new tserver with Master: 509739b6da1343d19d9b7489502b8093 (127.7.146.193:35411)
I20260619 06:40:01.798681 10830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:36143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:40:01.816151 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43955
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:02.135118 11025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:02.135406 11025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:02.135499 11025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:02.142858 11025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:02.142971 11025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:02.143091 11025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:40:02.152758 11025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:02.153965 11025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:02.155191 11025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:02.168434 11030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:02.168402 11033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:02.168495 11031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:02.169664 11025 server_base.cc:1034] running on GCE node
I20260619 06:40:02.170451 11025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:02.171723 11025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:02.172966 11025 hybrid_clock.cc:648] HybridClock initialized: now 1781851202172869 us; error 91 us; skew 500 ppm
I20260619 06:40:02.173276 11025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:02 dist-test-slave-qfk4 krb5kdc[10785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851202, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:02.176370 11025 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:40:02.177909 11025 webserver.cc:460] Webserver started at http://127.7.146.194:45985/ using document root <none> and password file <none>
I20260619 06:40:02.178580 11025 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:02.178684 11025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:02.178972 11025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:02.181888 11025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "25b4aad94aa349f9b4e37a56f1185d85"
format_stamp: "Formatted at 2026-06-19 06:40:02 on dist-test-slave-qfk4"
server_key: "eb2cd62584361cb2065bc629557c7308"
server_key_iv: "64b2314f486f47acb110f5303792a2ed"
server_key_version: "encryptionkey@0"
I20260619 06:40:02.182624 11025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "25b4aad94aa349f9b4e37a56f1185d85"
format_stamp: "Formatted at 2026-06-19 06:40:02 on dist-test-slave-qfk4"
server_key: "eb2cd62584361cb2065bc629557c7308"
server_key_iv: "64b2314f486f47acb110f5303792a2ed"
server_key_version: "encryptionkey@0"
I20260619 06:40:02.187886 11025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:40:02.191463 11040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:02.192966 11025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:40:02.193117 11025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "25b4aad94aa349f9b4e37a56f1185d85"
format_stamp: "Formatted at 2026-06-19 06:40:02 on dist-test-slave-qfk4"
server_key: "eb2cd62584361cb2065bc629557c7308"
server_key_iv: "64b2314f486f47acb110f5303792a2ed"
server_key_version: "encryptionkey@0"
I20260619 06:40:02.193296 11025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:02.240458 11025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:02.245077 11025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:02.245425 11025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:02.246752 11025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:02.248253 11025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:02.248329 11025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:02.248418 11025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:02.248459 11025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:02.277189 11025 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:40043
I20260619 06:40:02.277192 11152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:40043 every 8 connection(s)
I20260619 06:40:02.278471 11025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:40:02.283362 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 11025
I20260619 06:40:02.283545 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:40:02.284065 7755 external_mini_cluster.cc:1428] Setting key c106fc0fae1c36982c71ec037f565922
Jun 19 06:40:02 dist-test-slave-qfk4 krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851202, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:02.298970 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:02.280907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:42105 (local address 127.7.146.254:43955)
0619 06:40:02.281104 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:02.281110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:02.282326 (+ 1216us) server_negotiation.cc:407] Connection header received
0619 06:40:02.284557 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:02.284589 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:02.284679 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:02.284820 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:02.286433 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.287143 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.288683 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.289134 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.289255 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:02.292824 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:02.292830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:02.292835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:02.292872 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:02.295800 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.296513 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.296519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.296522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.296692 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.297140 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.297145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.297149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.297534 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:02.297671 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:02.298381 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 06:40:02.298613 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:40:02.299772 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:02.281481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:43955 (local address 127.7.146.194:42105)
0619 06:40:02.282166 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:02.282212 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:02.284190 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:02.285093 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:02.285106 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:02.285631 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:02.286272 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:02.286290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.287289 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.287293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:02.288515 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:02.288526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.289365 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.289373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:02.289598 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:02.290233 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:02.290257 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:02.292550 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:02.295954 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:02.295961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:02.295974 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:02.296406 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:02.296787 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:02.296790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:02.296793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:02.297028 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:02.297677 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:02.297685 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:02.298038 (+ 353us) client_negotiation.cc:769] Sending connection context
0619 06:40:02.298272 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 06:40:02.298659 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":295,"threads_started":1}
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
I20260619 06:40:02.301175 11153 heartbeater.cc:347] Connected to a master server at 127.7.146.254:43955
I20260619 06:40:02.301483 11153 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:02.302224 11153 heartbeater.cc:511] Master 127.7.146.254:43955 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
I20260619 06:40:02.303685 10830 ts_manager.cc:194] Registered new tserver with Master: 25b4aad94aa349f9b4e37a56f1185d85 (127.7.146.194:40043)
I20260619 06:40:02.304545 10830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:42105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:40:02.324708 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43955
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:02.635154 11158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:02.635423 11158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:02.635536 11158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:02.645705 11158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:02.645859 11158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:02.646049 11158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:40:02.661765 11158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:02.663472 11158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:02.665355 11158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:02.680665 11166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:02.680593 11164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:02.681715 11163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:02.681710 11158 server_base.cc:1034] running on GCE node
I20260619 06:40:02.682533 11158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:02.683481 11158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:02.684666 11158 hybrid_clock.cc:648] HybridClock initialized: now 1781851202684598 us; error 60 us; skew 500 ppm
I20260619 06:40:02.684988 11158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:02 dist-test-slave-qfk4 krb5kdc[10785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851202, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:02.688050 11158 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:40:02.689520 11158 webserver.cc:460] Webserver started at http://127.7.146.195:40823/ using document root <none> and password file <none>
I20260619 06:40:02.690171 11158 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:02.690285 11158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:02.690573 11158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:02.693542 11158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "2bab83375f56424ca4bc7ac84b969c5f"
format_stamp: "Formatted at 2026-06-19 06:40:02 on dist-test-slave-qfk4"
server_key: "45b052dc6ffb612aa16511c70b0a67e6"
server_key_iv: "e91881410f373cbb63a4474798ecc624"
server_key_version: "encryptionkey@0"
I20260619 06:40:02.694379 11158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "2bab83375f56424ca4bc7ac84b969c5f"
format_stamp: "Formatted at 2026-06-19 06:40:02 on dist-test-slave-qfk4"
server_key: "45b052dc6ffb612aa16511c70b0a67e6"
server_key_iv: "e91881410f373cbb63a4474798ecc624"
server_key_version: "encryptionkey@0"
I20260619 06:40:02.699729 11158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:40:02.703454 11173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:02.705228 11158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:02.705403 11158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "2bab83375f56424ca4bc7ac84b969c5f"
format_stamp: "Formatted at 2026-06-19 06:40:02 on dist-test-slave-qfk4"
server_key: "45b052dc6ffb612aa16511c70b0a67e6"
server_key_iv: "e91881410f373cbb63a4474798ecc624"
server_key_version: "encryptionkey@0"
I20260619 06:40:02.705611 11158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:02.719179 11158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:02.723446 11158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:02.723703 11158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:02.724784 11158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:02.726078 11158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:02.726151 11158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:02.726264 11158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:02.726315 11158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:02.758554 11158 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:37457
I20260619 06:40:02.758574 11285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:37457 every 8 connection(s)
I20260619 06:40:02.759932 11158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:40:02.765769 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 11158
I20260619 06:40:02.765964 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:40:02.766531 7755 external_mini_cluster.cc:1428] Setting key 6f9a78f645d14b008b4f3bed21204dcc
Jun 19 06:40:02 dist-test-slave-qfk4 krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851202, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:02.781095 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:02.762877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:44273 (local address 127.7.146.254:43955)
0619 06:40:02.763096 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:02.763102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:02.764485 (+ 1383us) server_negotiation.cc:407] Connection header received
0619 06:40:02.766536 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:02.766541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:02.766630 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:02.766722 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:02.768531 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.769646 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.771174 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.771567 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.771675 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:02.775257 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:02.775262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:02.775265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:02.775297 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:02.777712 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.778444 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.778448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.778450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.778584 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.778992 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.778998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.779001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.779281 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:02.779408 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:02.780449 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0619 06:40:02.780732 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 06:40:02.781921 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:02.763437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:43955 (local address 127.7.146.195:44273)
0619 06:40:02.764295 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:02.764342 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:02.766210 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:02.766990 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:02.766999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:02.767660 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:02.768360 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:02.768386 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.769780 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.769786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:02.771036 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:02.771048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.771780 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.771787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:02.772023 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:02.772692 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:02.772715 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:02.775043 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:02.777864 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:02.777873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:02.777901 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:02.778330 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:02.778690 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:02.778694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:02.778698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:02.778887 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:02.779402 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:02.779409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:02.779737 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 06:40:02.779986 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 06:40:02.780468 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":383,"threads_started":1}
I20260619 06:40:02.783540 11286 heartbeater.cc:347] Connected to a master server at 127.7.146.254:43955
I20260619 06:40:02.783926 11286 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:02.784900 11286 heartbeater.cc:511] Master 127.7.146.254:43955 requested a full tablet report, sending...
I20260619 06:40:02.786541 10830 ts_manager.cc:194] Registered new tserver with Master: 2bab83375f56424ca4bc7ac84b969c5f (127.7.146.195:37457)
I20260619 06:40:02.787380 10830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:44273
I20260619 06:40:02.794847 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:40:02.801812 11019 heartbeater.cc:503] Master 127.7.146.254:43955 was elected leader, sending a full tablet report...
Jun 19 06:40:02 dist-test-slave-qfk4 krb5kdc[10785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851202, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 06:40:02 dist-test-slave-qfk4 krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851202, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:02.842846 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:02.826032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.7.146.254:43955)
0619 06:40:02.826253 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:02.826268 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:02.826638 (+ 370us) server_negotiation.cc:407] Connection header received
0619 06:40:02.826864 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:02.826869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:02.826935 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:02.827025 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:02.828792 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.829521 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.831384 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.831818 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.831916 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:02.836512 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:02.836520 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:02.836525 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:02.836600 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:02.839954 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.840677 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.840684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.840687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.840857 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.841252 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.841257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.841260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.841715 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:02.841952 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:02.842262 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 06:40:02.842505 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 06:40:02.843538 10830 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58296
I20260619 06:40:02.845178 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 10891
Jun 19 06:40:02 dist-test-slave-qfk4 krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851202, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:40:02.865113 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:02.846818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.7.146.193:35411)
0619 06:40:02.847445 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:02.847455 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:02.847483 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:40:02.847579 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:02.847588 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:02.847870 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:02.848059 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:02.849228 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.850945 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.852252 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.853125 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.853227 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:02.858043 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:02.858060 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:02.858083 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:02.858139 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:02.862039 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.862802 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.862811 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.862814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.862965 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.863411 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.863416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.863420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.863862 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:02.864163 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:02.864411 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 06:40:02.864721 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":286,"threads_started":1}
W20260619 06:40:02.866425 11014 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39342
W20260619 06:40:02.867544 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 10891: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:40:02.867734 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 10891
I20260619 06:40:02.879946 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 11025
Jun 19 06:40:02 dist-test-slave-qfk4 krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851202, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:40:02.894989 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:02.880820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.7.146.194:40043)
0619 06:40:02.881327 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:02.881336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:02.881366 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:40:02.881429 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:02.881435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:02.881637 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:02.881762 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:02.882948 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.883949 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.885346 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.885929 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.886024 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:02.889849 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:02.889861 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:02.889876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:02.889911 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:02.892414 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.893130 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.893137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.893139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.893261 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.893581 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.893585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.893587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.893967 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:02.894195 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:02.894475 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 06:40:02.894686 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":267,"threads_started":1}
W20260619 06:40:02.896255 11148 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47616
W20260619 06:40:02.897621 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 11025: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:40:02.897773 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 11025
I20260619 06:40:02.909343 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 11158
Jun 19 06:40:02 dist-test-slave-qfk4 krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851202, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:40:02.925537 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:02.910249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.7.146.195:37457)
0619 06:40:02.910906 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:02.910912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:02.910941 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:40:02.910999 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:02.911005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:02.911209 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:02.911346 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:02.912685 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.913789 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.915032 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:02.915599 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:02.915696 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:02.920161 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:02.920173 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:02.920188 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:02.920232 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:02.922961 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.923721 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.923728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.923730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.923865 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:02.924173 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:02.924176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:02.924178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:02.924520 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:02.924798 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:02.925056 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 06:40:02.925249 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":301,"threads_started":1}
W20260619 06:40:02.926818 11281 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37060
W20260619 06:40:02.927868 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 11158: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:40:02.928030 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 11158
I20260619 06:40:02.939284 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 10798
I20260619 06:40:02.940506 10860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:03.041095 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 10798
2026-06-19T06:40:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3427 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:40:03 dist-test-slave-qfk4 krb5kdc[11303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:40:03 dist-test-slave-qfk4 krb5kdc[11303](info): set up 2 sockets
Jun 19 06:40:03 dist-test-slave-qfk4 krb5kdc[11303](info): commencing operation
krb5kdc: starting...
W20260619 06:40:04.036986 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:40:04 dist-test-slave-qfk4 krb5kdc[11303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:40:04.220415 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34901
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:34901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:04.532408 11316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:04.532725 11316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:04.532838 11316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:04.539999 11316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:04.540119 11316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:04.540174 11316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:04.540249 11316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:04.540302 11316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:04.550262 11316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:34901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:04.551600 11316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:04.552954 11316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:04.564378 11322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:04.564438 11324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:04.564601 11321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:04.565493 11316 server_base.cc:1034] running on GCE node
I20260619 06:40:04.566278 11316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:04.567757 11316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:04.568981 11316 hybrid_clock.cc:648] HybridClock initialized: now 1781851204568924 us; error 43 us; skew 500 ppm
I20260619 06:40:04.569307 11316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:04 dist-test-slave-qfk4 krb5kdc[11303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851204, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:04.572407 11316 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:40:04.573940 11316 webserver.cc:460] Webserver started at http://127.7.146.254:33777/ using document root <none> and password file <none>
I20260619 06:40:04.574646 11316 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:04.574764 11316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:04.575064 11316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:04.578225 11316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "31fdabad68d941c2883957c3c5a08623"
format_stamp: "Formatted at 2026-06-19 06:40:04 on dist-test-slave-qfk4"
server_key: "3b985367c0761b855a615c6f1a7c6c09"
server_key_iv: "fc61b2319dd9847e43b6f60fda3e8f51"
server_key_version: "encryptionkey@0"
I20260619 06:40:04.579007 11316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "31fdabad68d941c2883957c3c5a08623"
format_stamp: "Formatted at 2026-06-19 06:40:04 on dist-test-slave-qfk4"
server_key: "3b985367c0761b855a615c6f1a7c6c09"
server_key_iv: "fc61b2319dd9847e43b6f60fda3e8f51"
server_key_version: "encryptionkey@0"
I20260619 06:40:04.584261 11316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:40:04.587749 11331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:04.589170 11316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:04.589345 11316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "31fdabad68d941c2883957c3c5a08623"
format_stamp: "Formatted at 2026-06-19 06:40:04 on dist-test-slave-qfk4"
server_key: "3b985367c0761b855a615c6f1a7c6c09"
server_key_iv: "fc61b2319dd9847e43b6f60fda3e8f51"
server_key_version: "encryptionkey@0"
I20260619 06:40:04.589533 11316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:04.603317 11316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:04.607635 11316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:04.607911 11316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:04.624311 11316 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:34901
I20260619 06:40:04.624330 11382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:34901 every 8 connection(s)
I20260619 06:40:04.625679 11316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:04.626371 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 11316
I20260619 06:40:04.626597 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:04.627372 7755 external_mini_cluster.cc:1428] Setting key 11b2794dea5c31af704b764530564623
I20260619 06:40:04.630275 11383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:40:04 dist-test-slave-qfk4 krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:04.640677 11383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623: Bootstrap starting.
I20260619 06:40:04.644011 11383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:04.645058 11383 log.cc:824] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:04.648180 11383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623: No bootstrap required, opened a new log
I20260619 06:40:04.648837 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:04.629454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.7.146.254:34901)
0619 06:40:04.630237 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:04.630249 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:04.630293 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:40:04.631891 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:04.631913 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:04.632275 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:04.632678 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:04.634114 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:04.635454 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:04.636705 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:04.637394 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:04.637591 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:04.641455 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:04.641463 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:04.641478 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:04.641520 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:04.644632 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:04.645337 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:04.645344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:04.645351 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:04.645552 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:04.645961 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:04.645970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:04.645973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:04.646607 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:04.646825 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:04.647351 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 06:40:04.647704 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":396,"threads_started":1}
I20260619 06:40:04.655596 11383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fdabad68d941c2883957c3c5a08623" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34901 } }
I20260619 06:40:04.655965 11383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:04.656064 11383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31fdabad68d941c2883957c3c5a08623, State: Initialized, Role: FOLLOWER
I20260619 06:40:04.656620 11383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fdabad68d941c2883957c3c5a08623" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34901 } }
I20260619 06:40:04.656834 11383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:04.656952 11383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:04.657107 11383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:04.659176 11383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fdabad68d941c2883957c3c5a08623" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34901 } }
I20260619 06:40:04.659653 11383 leader_election.cc:302] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31fdabad68d941c2883957c3c5a08623; no voters:
I20260619 06:40:04.660236 11383 leader_election.cc:288] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:04.660396 11388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:04.661593 11388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [term 1 LEADER]: Becoming Leader. State: Replica: 31fdabad68d941c2883957c3c5a08623, State: Running, Role: LEADER
I20260619 06:40:04.662135 11388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fdabad68d941c2883957c3c5a08623" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34901 } }
I20260619 06:40:04.662811 11383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:04.665002 11389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31fdabad68d941c2883957c3c5a08623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fdabad68d941c2883957c3c5a08623" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34901 } } }
I20260619 06:40:04.665510 11389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:04.665360 11390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31fdabad68d941c2883957c3c5a08623. Latest consensus state: current_term: 1 leader_uuid: "31fdabad68d941c2883957c3c5a08623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fdabad68d941c2883957c3c5a08623" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34901 } } }
I20260619 06:40:04.665723 11390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:04.667649 11396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:04.672709 11396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:04.682312 11396 catalog_manager.cc:1347] Generated new cluster ID: 93ce65a73fa448edac3202f31833b3bc
I20260619 06:40:04.682456 11396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:04.691701 11396 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:04.693203 11396 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:04.699927 11396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31fdabad68d941c2883957c3c5a08623: Generated new TSK 0
I20260619 06:40:04.700803 11396 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:40:04.746856 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34901
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:05.060554 11409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:05.060880 11409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:05.060998 11409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:05.068463 11409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:05.068595 11409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:05.068761 11409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:40:05.078346 11409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:05.079576 11409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:05.080931 11409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:05.093720 11417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:05.093758 11415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:05.094024 11414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:05.094777 11409 server_base.cc:1034] running on GCE node
I20260619 06:40:05.095407 11409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:05.096378 11409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:05.097577 11409 hybrid_clock.cc:648] HybridClock initialized: now 1781851205097479 us; error 98 us; skew 500 ppm
I20260619 06:40:05.097965 11409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:05 dist-test-slave-qfk4 krb5kdc[11303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851205, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:05.101164 11409 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:40:05.102699 11409 webserver.cc:460] Webserver started at http://127.7.146.193:41653/ using document root <none> and password file <none>
I20260619 06:40:05.103461 11409 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:05.103588 11409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:05.103886 11409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:05.106956 11409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "47919b8c47cf413ab36d010a8c25d92e"
format_stamp: "Formatted at 2026-06-19 06:40:05 on dist-test-slave-qfk4"
server_key: "1801ecd319e6a3c706032ef6bc571ef8"
server_key_iv: "afa7471153f59cf940890d0ba7ba6681"
server_key_version: "encryptionkey@0"
I20260619 06:40:05.108083 11409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "47919b8c47cf413ab36d010a8c25d92e"
format_stamp: "Formatted at 2026-06-19 06:40:05 on dist-test-slave-qfk4"
server_key: "1801ecd319e6a3c706032ef6bc571ef8"
server_key_iv: "afa7471153f59cf940890d0ba7ba6681"
server_key_version: "encryptionkey@0"
I20260619 06:40:05.113643 11409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 06:40:05.117359 11424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:05.118744 11409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:40:05.118903 11409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "47919b8c47cf413ab36d010a8c25d92e"
format_stamp: "Formatted at 2026-06-19 06:40:05 on dist-test-slave-qfk4"
server_key: "1801ecd319e6a3c706032ef6bc571ef8"
server_key_iv: "afa7471153f59cf940890d0ba7ba6681"
server_key_version: "encryptionkey@0"
I20260619 06:40:05.119122 11409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:05.137303 11409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:05.141721 11409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:05.141991 11409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:05.142930 11409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:05.144265 11409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:05.144354 11409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:05.144471 11409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:05.144523 11409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:05.174049 11409 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:44909
I20260619 06:40:05.174117 11536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:44909 every 8 connection(s)
I20260619 06:40:05.175333 11409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:40:05.176478 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 11409
I20260619 06:40:05.176726 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:40:05.177500 7755 external_mini_cluster.cc:1428] Setting key 322bc6f933cc89ed2c2904dc967d34d2
Jun 19 06:40:05 dist-test-slave-qfk4 krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851205, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
I20260619 06:40:05.197578 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:05.178193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:49121 (local address 127.7.146.254:34901)
0619 06:40:05.179147 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:05.179154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:05.179720 (+ 566us) server_negotiation.cc:407] Connection header received
0619 06:40:05.182245 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:05.182250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:05.182317 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:05.182406 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:05.184369 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:05.185115 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:05.186663 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:05.187285 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:05.187405 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:05.190912 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:05.190918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:05.190923 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:05.190962 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:05.194377 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:05.195233 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:05.195239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:05.195242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:05.195380 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:05.195799 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:05.195804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:05.195807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:05.196113 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:05.196225 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:05.197103 (+ 878us) server_negotiation.cc:299] Negotiation successful
0619 06:40:05.197275 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":214,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
I20260619 06:40:05.198818 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:05.178801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34901 (local address 127.7.146.193:49121)
0619 06:40:05.179552 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:05.179607 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:05.181955 (+ 2348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:05.182686 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:05.182697 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:05.183404 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:05.184197 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:05.184221 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:05.185267 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:05.185273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:05.186513 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:05.186526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:05.187506 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:05.187510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:05.187714 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:05.188374 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:05.188401 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:05.190640 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:05.194554 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:05.194562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:05.194577 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:05.195102 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:05.195478 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:05.195481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:05.195483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:05.195693 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:05.196249 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:05.196257 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:05.196709 (+ 452us) client_negotiation.cc:769] Sending connection context
0619 06:40:05.197034 (+ 325us) client_negotiation.cc:240] Negotiation successful
0619 06:40:05.197526 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":301,"threads_started":1}
I20260619 06:40:05.200588 11537 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34901
I20260619 06:40:05.200942 11537 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:05.201668 11537 heartbeater.cc:511] Master 127.7.146.254:34901 requested a full tablet report, sending...
I20260619 06:40:05.203781 11348 ts_manager.cc:194] Registered new tserver with Master: 47919b8c47cf413ab36d010a8c25d92e (127.7.146.193:44909)
I20260619 06:40:05.205276 11348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:49121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:40:05.219893 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34901
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:05.528793 11543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:05.529099 11543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:05.529232 11543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:05.536964 11543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:05.537096 11543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:05.537252 11543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:40:05.547184 11543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:05.548466 11543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:05.549762 11543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:05.561985 11548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:05.562072 11551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:05.562264 11549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:05.562851 11543 server_base.cc:1034] running on GCE node
I20260619 06:40:05.563477 11543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:05.564443 11543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:05.565656 11543 hybrid_clock.cc:648] HybridClock initialized: now 1781851205565603 us; error 48 us; skew 500 ppm
I20260619 06:40:05.565974 11543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:05 dist-test-slave-qfk4 krb5kdc[11303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851205, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:05.569079 11543 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:40:05.570482 11543 webserver.cc:460] Webserver started at http://127.7.146.194:38763/ using document root <none> and password file <none>
I20260619 06:40:05.571144 11543 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:05.571262 11543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:05.571569 11543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:05.574589 11543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "e9d8e9c871734e15b86a0e29672869c6"
format_stamp: "Formatted at 2026-06-19 06:40:05 on dist-test-slave-qfk4"
server_key: "62f57a59229620fdf3d49cd7eadde9c8"
server_key_iv: "77fe6ec62d70d06c81738b3728e5563e"
server_key_version: "encryptionkey@0"
I20260619 06:40:05.575409 11543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "e9d8e9c871734e15b86a0e29672869c6"
format_stamp: "Formatted at 2026-06-19 06:40:05 on dist-test-slave-qfk4"
server_key: "62f57a59229620fdf3d49cd7eadde9c8"
server_key_iv: "77fe6ec62d70d06c81738b3728e5563e"
server_key_version: "encryptionkey@0"
I20260619 06:40:05.580855 11543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:40:05.584539 11558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:05.586066 11543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:05.586225 11543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "e9d8e9c871734e15b86a0e29672869c6"
format_stamp: "Formatted at 2026-06-19 06:40:05 on dist-test-slave-qfk4"
server_key: "62f57a59229620fdf3d49cd7eadde9c8"
server_key_iv: "77fe6ec62d70d06c81738b3728e5563e"
server_key_version: "encryptionkey@0"
I20260619 06:40:05.586417 11543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:05.606366 11543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:05.610713 11543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:05.611004 11543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:05.612063 11543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:05.613448 11543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:05.613529 11543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:05.613651 11543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:05.613703 11543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:05.642702 11543 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:33023
I20260619 06:40:05.642721 11670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:33023 every 8 connection(s)
I20260619 06:40:05.644271 11543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:40:05.646628 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 11543
I20260619 06:40:05.646828 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:40:05.647264 7755 external_mini_cluster.cc:1428] Setting key 48df507308bc0ad7d9feb6fdc0f7c3e2
Jun 19 06:40:05 dist-test-slave-qfk4 krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851205, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
I20260619 06:40:05.663688 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:05.646901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:58865 (local address 127.7.146.254:34901)
0619 06:40:05.647088 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:05.647094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:05.648336 (+ 1242us) server_negotiation.cc:407] Connection header received
0619 06:40:05.650232 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:05.650238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:05.650316 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:05.650397 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:05.651902 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:05.652667 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:05.654162 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:05.654580 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:05.654691 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:05.658210 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:05.658215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:05.658218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:05.658251 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:05.660673 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:05.661540 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:05.661544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:05.661546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:05.661669 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:05.662112 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:05.662115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:05.662117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:05.662347 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:05.662480 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:05.663200 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 06:40:05.663366 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:40:05.664731 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:05.647406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34901 (local address 127.7.146.194:58865)
0619 06:40:05.648186 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:05.648231 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:05.649969 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:05.650649 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:05.650658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:05.651161 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:05.651736 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:05.651754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:05.652808 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:05.652812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:05.654028 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:05.654040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:05.654784 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:05.654791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:05.655007 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:05.655633 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:05.655658 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:05.657985 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:05.660813 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:05.660822 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:05.660840 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:05.661431 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:05.661767 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:05.661772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:05.661776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:05.662021 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:05.662498 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:05.662513 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:05.662893 (+ 380us) client_negotiation.cc:769] Sending connection context
0619 06:40:05.663121 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 06:40:05.663567 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":314,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
I20260619 06:40:05.666388 11671 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34901
I20260619 06:40:05.666697 11671 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:05.667481 11671 heartbeater.cc:511] Master 127.7.146.254:34901 requested a full tablet report, sending...
I20260619 06:40:05.669035 11348 ts_manager.cc:194] Registered new tserver with Master: e9d8e9c871734e15b86a0e29672869c6 (127.7.146.194:33023)
I20260619 06:40:05.670060 11348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:58865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:40:05.687893 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34901
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:06.000998 11676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:06.001307 11676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:06.001431 11676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:06.008728 11676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:06.008867 11676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:06.009034 11676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:40:06.018963 11676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:06.020248 11676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:06.021628 11676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:06.034500 11682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:06.034529 11684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:06.034565 11681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:06.035324 11676 server_base.cc:1034] running on GCE node
I20260619 06:40:06.035935 11676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:06.037006 11676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:06.038220 11676 hybrid_clock.cc:648] HybridClock initialized: now 1781851206038169 us; error 42 us; skew 500 ppm
I20260619 06:40:06.038539 11676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:06 dist-test-slave-qfk4 krb5kdc[11303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851206, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:06.041601 11676 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:40:06.043080 11676 webserver.cc:460] Webserver started at http://127.7.146.195:34563/ using document root <none> and password file <none>
I20260619 06:40:06.043743 11676 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:06.043861 11676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:06.044155 11676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:06.047072 11676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "415a86408d284a0f8555ec0d73f95f85"
format_stamp: "Formatted at 2026-06-19 06:40:06 on dist-test-slave-qfk4"
server_key: "48a5308e9349c3a7b6e5dd3bed456210"
server_key_iv: "3c93ed0192b23ff80c67da801598b6c3"
server_key_version: "encryptionkey@0"
I20260619 06:40:06.047827 11676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "415a86408d284a0f8555ec0d73f95f85"
format_stamp: "Formatted at 2026-06-19 06:40:06 on dist-test-slave-qfk4"
server_key: "48a5308e9349c3a7b6e5dd3bed456210"
server_key_iv: "3c93ed0192b23ff80c67da801598b6c3"
server_key_version: "encryptionkey@0"
I20260619 06:40:06.052970 11676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 06:40:06.056370 11691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:06.057704 11676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:06.057847 11676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "415a86408d284a0f8555ec0d73f95f85"
format_stamp: "Formatted at 2026-06-19 06:40:06 on dist-test-slave-qfk4"
server_key: "48a5308e9349c3a7b6e5dd3bed456210"
server_key_iv: "3c93ed0192b23ff80c67da801598b6c3"
server_key_version: "encryptionkey@0"
I20260619 06:40:06.058028 11676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:06.075798 11676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:06.079797 11676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:06.080070 11676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:06.081059 11676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:06.082351 11676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:06.082427 11676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:06.082548 11676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:06.082598 11676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:06.113514 11676 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:39337
I20260619 06:40:06.113551 11803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:39337 every 8 connection(s)
I20260619 06:40:06.114758 11676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:40:06.115272 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 11676
I20260619 06:40:06.115447 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:40:06.116053 7755 external_mini_cluster.cc:1428] Setting key 628f1aa4b963e98d9ccff711c76f483a
Jun 19 06:40:06 dist-test-slave-qfk4 krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851206, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:06.136602 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:06.117571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:52687 (local address 127.7.146.254:34901)
0619 06:40:06.117730 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:06.117735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:06.119368 (+ 1633us) server_negotiation.cc:407] Connection header received
0619 06:40:06.121487 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:06.121491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:06.121559 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:06.121642 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:06.123520 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.124591 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.126511 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.126929 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.127051 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:06.130745 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:06.130752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:06.130758 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:06.130805 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:06.133732 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.134405 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.134408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.134410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.134547 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.135065 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.135067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.135069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.135316 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:06.135418 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:06.136143 (+ 725us) server_negotiation.cc:299] Negotiation successful
0619 06:40:06.136305 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:40:06.137691 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:06.118337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34901 (local address 127.7.146.195:52687)
0619 06:40:06.119218 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:06.119258 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:06.121197 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:06.121906 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:06.121915 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:06.122458 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:06.123319 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:06.123347 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.124853 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.124859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:06.126360 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:06.126373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.127159 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.127166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:06.127429 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:06.128039 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:06.128061 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:06.130486 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:06.133866 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:06.133873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:06.133888 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:06.134314 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:06.134671 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:06.134676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:06.134678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:06.134969 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:06.135427 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:06.135433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:06.135825 (+ 392us) client_negotiation.cc:769] Sending connection context
0619 06:40:06.136090 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 06:40:06.136518 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":322,"threads_started":1}
I20260619 06:40:06.139110 11804 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34901
I20260619 06:40:06.139374 11804 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:06.140043 11804 heartbeater.cc:511] Master 127.7.146.254:34901 requested a full tablet report, sending...
I20260619 06:40:06.141471 11348 ts_manager.cc:194] Registered new tserver with Master: 415a86408d284a0f8555ec0d73f95f85 (127.7.146.195:39337)
I20260619 06:40:06.142263 11348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:52687
I20260619 06:40:06.144214 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:40:06 dist-test-slave-qfk4 krb5kdc[11303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:40:06 dist-test-slave-qfk4 krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:40:06.195487 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:06.177454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.7.146.193:44909)
0619 06:40:06.178106 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:06.178116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:06.178485 (+ 369us) server_negotiation.cc:407] Connection header received
0619 06:40:06.178656 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:06.178664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:06.178870 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:06.179006 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:06.180338 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.181400 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.182646 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.183176 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.183296 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:06.190050 (+ 6754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:06.190063 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:06.190079 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:06.190144 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:06.192966 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.193720 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.193728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.193731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.193873 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.194174 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.194177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.194179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.194519 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:06.194751 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:06.194978 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 06:40:06.195171 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":359,"threads_started":1}
Jun 19 06:40:06 dist-test-slave-qfk4 krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:06.208673 11537 heartbeater.cc:503] Master 127.7.146.254:34901 was elected leader, sending a full tablet report...
I20260619 06:40:06.213214 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:06.199618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60980 (local address 127.7.146.254:34901)
0619 06:40:06.199814 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:06.199820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:06.199980 (+ 160us) server_negotiation.cc:407] Connection header received
0619 06:40:06.200103 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:06.200107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:06.200168 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:06.200244 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:06.201587 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.202292 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.203516 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.203949 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.204044 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:06.207713 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:06.207718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:06.207721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:06.207754 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:06.210792 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.211401 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.211404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.211406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.211546 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.211930 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.211933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.211935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.212247 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:06.212411 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:06.212792 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 06:40:06.212957 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:40:06.213748 11348 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60980
I20260619 06:40:06.214715 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 11409
I20260619 06:40:06.227089 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:06.215551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.7.146.193:44909)
0619 06:40:06.215736 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:06.215742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:06.215912 (+ 170us) server_negotiation.cc:407] Connection header received
0619 06:40:06.216066 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:06.216071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:06.216132 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:06.216211 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:06.217350 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.218014 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.219246 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.219669 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.219782 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:06.222001 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:06.222007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:06.222010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:06.222041 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:06.224826 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.225401 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.225405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.225407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.225520 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.225837 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.225841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.225845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.226087 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:06.226294 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:06.226593 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 06:40:06.226790 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:40:06.227669 11532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:06.308977 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 11409
I20260619 06:40:06.321344 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 11543
Jun 19 06:40:06 dist-test-slave-qfk4 krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:40:06.336799 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:06.322325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.7.146.194:33023)
0619 06:40:06.322896 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:06.322903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:06.322928 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:40:06.322994 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:06.323001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:06.323199 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:06.323334 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:06.324454 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.325546 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.326748 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.327347 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.327446 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:06.331281 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:06.331292 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:06.331311 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:06.331356 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:06.334136 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.334785 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.334792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.334795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.334917 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.335266 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.335270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.335272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.335608 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:06.335885 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:06.336175 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 06:40:06.336428 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":296,"threads_started":1}
I20260619 06:40:06.338042 11666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:06.420903 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 11543
I20260619 06:40:06.432829 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 11676
Jun 19 06:40:06 dist-test-slave-qfk4 krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:40:06.448700 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:06.433653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.7.146.195:39337)
0619 06:40:06.434284 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:06.434291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:06.434320 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:40:06.434383 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:06.434389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:06.434597 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:06.434726 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:06.435908 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.436991 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.438322 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:06.438895 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:06.438999 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:06.443197 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:06.443211 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:06.443226 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:06.443264 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:06.445932 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.446606 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.446613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.446615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.446740 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:06.447109 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:06.447112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:06.447114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:06.447454 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:06.447762 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:06.448035 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 06:40:06.448299 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":339,"threads_started":1}
I20260619 06:40:06.449975 11799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:06.531826 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 11676
I20260619 06:40:06.543658 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 11316
I20260619 06:40:06.544953 11378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:06.646001 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 11316
2026-06-19T06:40:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3604 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:40:06 dist-test-slave-qfk4 krb5kdc[11823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:40:06 dist-test-slave-qfk4 krb5kdc[11823](info): set up 2 sockets
Jun 19 06:40:06 dist-test-slave-qfk4 krb5kdc[11823](info): commencing operation
krb5kdc: starting...
W20260619 06:40:07.640864 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:40:07 dist-test-slave-qfk4 krb5kdc[11823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:40:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:40:07.826256 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:43781
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:43781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:08.137643 11836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:08.137974 11836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:08.138120 11836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:08.145545 11836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:08.145673 11836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:08.145751 11836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:08.145824 11836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:08.145897 11836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:08.155946 11836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:43781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:43781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:08.157341 11836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:08.158643 11836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:08.171555 11841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:08.171506 11842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:08.171506 11844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:08.173425 11836 server_base.cc:1034] running on GCE node
I20260619 06:40:08.174270 11836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:08.175953 11836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:08.177187 11836 hybrid_clock.cc:648] HybridClock initialized: now 1781851208177136 us; error 49 us; skew 500 ppm
I20260619 06:40:08.177579 11836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:08 dist-test-slave-qfk4 krb5kdc[11823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851208, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:08.180920 11836 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:40:08.182076 11836 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:08.182210 11836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:08.182536 11836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:08.185643 11836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "cbc9200892de4eb1ae719a64b034d422"
format_stamp: "Formatted at 2026-06-19 06:40:08 on dist-test-slave-qfk4"
server_key: "4c726a4d0816ff9b2f53877d3476c506"
server_key_iv: "04f5f96462247f3aba6e0fbc05a292d6"
server_key_version: "encryptionkey@0"
I20260619 06:40:08.186414 11836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "cbc9200892de4eb1ae719a64b034d422"
format_stamp: "Formatted at 2026-06-19 06:40:08 on dist-test-slave-qfk4"
server_key: "4c726a4d0816ff9b2f53877d3476c506"
server_key_iv: "04f5f96462247f3aba6e0fbc05a292d6"
server_key_version: "encryptionkey@0"
I20260619 06:40:08.191545 11836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:40:08.195363 11850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:08.196976 11836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:08.197129 11836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "cbc9200892de4eb1ae719a64b034d422"
format_stamp: "Formatted at 2026-06-19 06:40:08 on dist-test-slave-qfk4"
server_key: "4c726a4d0816ff9b2f53877d3476c506"
server_key_iv: "04f5f96462247f3aba6e0fbc05a292d6"
server_key_version: "encryptionkey@0"
I20260619 06:40:08.197347 11836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:08.208616 11836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:08.212929 11836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:08.213177 11836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:08.229374 11836 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:43781
I20260619 06:40:08.229362 11901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:43781 every 8 connection(s)
I20260619 06:40:08.230322 11836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:08.233531 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 11836
I20260619 06:40:08.233768 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:08.234233 11902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:08.234544 7755 external_mini_cluster.cc:1428] Setting key 66584067223cd5b10579ad571e5cef2c
I20260619 06:40:08.248718 11902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422: Bootstrap starting.
Jun 19 06:40:08 dist-test-slave-qfk4 krb5kdc[11823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:08.252353 11902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:08.253564 11902 log.cc:824] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:08.256945 11902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422: No bootstrap required, opened a new log
I20260619 06:40:08.259078 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:08.236695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58626 (local address 127.7.146.254:43781)
0619 06:40:08.237653 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:08.237666 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:08.237709 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:40:08.239275 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:08.239300 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:08.239728 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:08.240103 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:08.242013 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:08.243399 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:08.245226 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:08.246042 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:08.246208 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:08.250913 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:08.250922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:08.250937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:08.250978 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:08.254200 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:08.255044 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:08.255052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:08.255060 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:08.255256 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:08.255759 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:08.255764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:08.255767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:08.256473 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:08.256772 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:08.257260 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 06:40:08.257728 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":416,"threads_started":1}
I20260619 06:40:08.265439 11902 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9200892de4eb1ae719a64b034d422" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43781 } }
I20260619 06:40:08.265782 11902 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:08.265861 11902 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc9200892de4eb1ae719a64b034d422, State: Initialized, Role: FOLLOWER
I20260619 06:40:08.266342 11902 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9200892de4eb1ae719a64b034d422" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43781 } }
I20260619 06:40:08.266525 11902 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:08.266639 11902 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:08.266775 11902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:08.269160 11902 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9200892de4eb1ae719a64b034d422" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43781 } }
I20260619 06:40:08.269616 11902 leader_election.cc:302] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbc9200892de4eb1ae719a64b034d422; no voters:
I20260619 06:40:08.270184 11902 leader_election.cc:288] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:08.270313 11907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:08.270920 11907 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [term 1 LEADER]: Becoming Leader. State: Replica: cbc9200892de4eb1ae719a64b034d422, State: Running, Role: LEADER
I20260619 06:40:08.271401 11907 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9200892de4eb1ae719a64b034d422" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43781 } }
I20260619 06:40:08.272048 11902 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:08.274463 11909 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc9200892de4eb1ae719a64b034d422. Latest consensus state: current_term: 1 leader_uuid: "cbc9200892de4eb1ae719a64b034d422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9200892de4eb1ae719a64b034d422" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43781 } } }
I20260619 06:40:08.274555 11908 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbc9200892de4eb1ae719a64b034d422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc9200892de4eb1ae719a64b034d422" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43781 } } }
I20260619 06:40:08.274953 11909 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:08.275043 11908 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:08.276662 11912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:08.281652 11912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:08.290812 11912 catalog_manager.cc:1347] Generated new cluster ID: 0bf7d5e1bcc84e0a848d7dafd011ca93
I20260619 06:40:08.290918 11912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:08.320978 11912 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:08.322623 11912 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:08.333575 11912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422: Generated new TSK 0
I20260619 06:40:08.334707 11912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:40:08.392338 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43781
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:08.729192 11928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:08.729504 11928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:08.729626 11928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:08.736799 11928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:08.736912 11928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:08.737066 11928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:40:08.746749 11928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:08.748075 11928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:08.749433 11928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:08.761790 11934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:08.762215 11933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:08.762292 11936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:08.762749 11928 server_base.cc:1034] running on GCE node
I20260619 06:40:08.763355 11928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:08.764339 11928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:08.765568 11928 hybrid_clock.cc:648] HybridClock initialized: now 1781851208765496 us; error 65 us; skew 500 ppm
I20260619 06:40:08.765906 11928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:08 dist-test-slave-qfk4 krb5kdc[11823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851208, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:08.768993 11928 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:40:08.770150 11928 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:08.770241 11928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:08.770509 11928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:08.773563 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "c873a59d34344ba7807dafbbb6f8dcf2"
format_stamp: "Formatted at 2026-06-19 06:40:08 on dist-test-slave-qfk4"
server_key: "8231ad83b161d771b5ab03a3c82b96c1"
server_key_iv: "ba7035d139c154d52dd9ae5ca16ef079"
server_key_version: "encryptionkey@0"
I20260619 06:40:08.774340 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "c873a59d34344ba7807dafbbb6f8dcf2"
format_stamp: "Formatted at 2026-06-19 06:40:08 on dist-test-slave-qfk4"
server_key: "8231ad83b161d771b5ab03a3c82b96c1"
server_key_iv: "ba7035d139c154d52dd9ae5ca16ef079"
server_key_version: "encryptionkey@0"
I20260619 06:40:08.779623 11928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:40:08.783346 11942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:08.784827 11928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:08.784987 11928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "c873a59d34344ba7807dafbbb6f8dcf2"
format_stamp: "Formatted at 2026-06-19 06:40:08 on dist-test-slave-qfk4"
server_key: "8231ad83b161d771b5ab03a3c82b96c1"
server_key_iv: "ba7035d139c154d52dd9ae5ca16ef079"
server_key_version: "encryptionkey@0"
I20260619 06:40:08.785197 11928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:08.807379 11928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:08.811687 11928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:08.811945 11928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:08.812964 11928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:08.814248 11928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:08.814337 11928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:08.814416 11928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:08.814466 11928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:08.844285 11928 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:46697
I20260619 06:40:08.844329 12054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:46697 every 8 connection(s)
I20260619 06:40:08.845161 11928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:40:08.852715 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 11928
I20260619 06:40:08.852909 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:40:08.853477 7755 external_mini_cluster.cc:1428] Setting key a81b87a99b4bfd5b9f812989e201bceb
Jun 19 06:40:08 dist-test-slave-qfk4 krb5kdc[11823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851208, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:08.864307 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:08.847542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:47861 (local address 127.7.146.254:43781)
0619 06:40:08.847935 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:08.847939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:08.848866 (+ 927us) server_negotiation.cc:407] Connection header received
0619 06:40:08.850662 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:08.850667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:08.850744 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:08.850838 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:08.852312 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:08.853122 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:08.854782 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:08.855281 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:08.855389 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:08.858958 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:08.858963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:08.858966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:08.859000 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:08.861433 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:08.862113 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:08.862118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:08.862121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:08.862232 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:08.862623 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:08.862628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:08.862632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:08.862907 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:08.863030 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:08.863715 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 06:40:08.863946 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":190,"threads_started":1}
I20260619 06:40:08.865324 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:08.847976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:43781 (local address 127.7.146.193:47861)
0619 06:40:08.848713 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:08.848758 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:08.850402 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:08.851069 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:08.851077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:08.851596 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:08.852151 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:08.852167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:08.853257 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:08.853261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:08.854627 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:08.854639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:08.855495 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:08.855502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:08.855705 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:08.856361 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:08.856413 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:08.858684 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:08.861562 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:08.861571 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:08.861589 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:08.862001 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:08.862329 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:08.862335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:08.862339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:08.862520 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:08.863029 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:08.863038 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:08.863356 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 06:40:08.863608 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 06:40:08.864137 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":272,"threads_started":1}
I20260619 06:40:08.866863 12055 heartbeater.cc:347] Connected to a master server at 127.7.146.254:43781
I20260619 06:40:08.867216 12055 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:08.868016 12055 heartbeater.cc:511] Master 127.7.146.254:43781 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
I20260619 06:40:08.870370 11867 ts_manager.cc:194] Registered new tserver with Master: c873a59d34344ba7807dafbbb6f8dcf2 (127.7.146.193:46697)
I20260619 06:40:08.871747 11867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:47861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:40:08.895051 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43781
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:09.203486 12061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:09.203863 12061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:09.204015 12061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:09.211299 12061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:09.211431 12061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:09.211601 12061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:40:09.222046 12061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:09.223371 12061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:09.224736 12061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:09.237593 12066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:09.237619 12067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:09.238283 12069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:09.238873 12061 server_base.cc:1034] running on GCE node
I20260619 06:40:09.239470 12061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:09.240465 12061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:09.241704 12061 hybrid_clock.cc:648] HybridClock initialized: now 1781851209241644 us; error 50 us; skew 500 ppm
I20260619 06:40:09.242085 12061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:09 dist-test-slave-qfk4 krb5kdc[11823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851209, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:09.245368 12061 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:40:09.246563 12061 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:09.246675 12061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:09.246989 12061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:09.250092 12061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "2c9be44f44cf437cb252e3da4865b0d3"
format_stamp: "Formatted at 2026-06-19 06:40:09 on dist-test-slave-qfk4"
server_key: "99ec56bcc1e8e97a8c2ebf84e828a443"
server_key_iv: "c2d27fe75d07ef8186936dd1b7d1bf37"
server_key_version: "encryptionkey@0"
I20260619 06:40:09.250942 12061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "2c9be44f44cf437cb252e3da4865b0d3"
format_stamp: "Formatted at 2026-06-19 06:40:09 on dist-test-slave-qfk4"
server_key: "99ec56bcc1e8e97a8c2ebf84e828a443"
server_key_iv: "c2d27fe75d07ef8186936dd1b7d1bf37"
server_key_version: "encryptionkey@0"
I20260619 06:40:09.256203 12061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 06:40:09.259748 12075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:09.261096 12061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:40:09.261247 12061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "2c9be44f44cf437cb252e3da4865b0d3"
format_stamp: "Formatted at 2026-06-19 06:40:09 on dist-test-slave-qfk4"
server_key: "99ec56bcc1e8e97a8c2ebf84e828a443"
server_key_iv: "c2d27fe75d07ef8186936dd1b7d1bf37"
server_key_version: "encryptionkey@0"
I20260619 06:40:09.261446 12061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:09.286496 12061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:09.290598 12061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:09.290865 12061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:09.291839 12061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:09.293161 12061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:09.293257 12061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:09.293361 12061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:09.293428 12061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:09.323285 12061 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:38615
I20260619 06:40:09.323345 12187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:38615 every 8 connection(s)
I20260619 06:40:09.324105 12061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:40:09.331840 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 12061
I20260619 06:40:09.332037 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:40:09.332533 7755 external_mini_cluster.cc:1428] Setting key b3c67c96ebc2c350a60495aec2028e69
Jun 19 06:40:09 dist-test-slave-qfk4 krb5kdc[11823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851209, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:09.343446 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:09.326505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:44313 (local address 127.7.146.254:43781)
0619 06:40:09.326667 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:09.326671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:09.327674 (+ 1003us) server_negotiation.cc:407] Connection header received
0619 06:40:09.329608 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:09.329615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:09.329694 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:09.329778 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:09.331259 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.332208 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.333963 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.334490 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.334639 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:09.337845 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:09.337851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:09.337855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:09.337894 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:09.340535 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:09.341390 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:09.341394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:09.341396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:09.341518 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:09.341924 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:09.341927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:09.341929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:09.342174 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:09.342308 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:09.343036 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 06:40:09.343199 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:40:09.344861 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:09.326854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:43781 (local address 127.7.146.194:44313)
0619 06:40:09.327515 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:09.327552 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:09.329333 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:09.330014 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:09.330023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:09.330530 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:09.331086 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:09.331103 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.332353 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.332357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:09.333797 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:09.333810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.334712 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.334717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:09.334905 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:09.335541 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:09.335563 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:09.337617 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:09.340665 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:09.340673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:09.340691 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:09.341247 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:09.341611 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:09.341615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:09.341618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:09.341824 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:09.342281 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:09.342289 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:09.342691 (+ 402us) client_negotiation.cc:769] Sending connection context
0619 06:40:09.342959 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 06:40:09.343470 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":267,"threads_started":1}
I20260619 06:40:09.346387 12188 heartbeater.cc:347] Connected to a master server at 127.7.146.254:43781
I20260619 06:40:09.346711 12188 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:09.347329 12188 heartbeater.cc:511] Master 127.7.146.254:43781 requested a full tablet report, sending...
I20260619 06:40:09.348758 11867 ts_manager.cc:194] Registered new tserver with Master: 2c9be44f44cf437cb252e3da4865b0d3 (127.7.146.194:38615)
I20260619 06:40:09.349530 11867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:44313
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:40:09.373883 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43781
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:09.687039 12193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:09.687325 12193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:09.687423 12193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:09.694777 12193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:09.694906 12193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:09.695031 12193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:40:09.704855 12193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:09.706110 12193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:09.707358 12193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:09.719839 12199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:09.719808 12198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:09.719892 12201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:09.720626 12193 server_base.cc:1034] running on GCE node
I20260619 06:40:09.721184 12193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:09.722057 12193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:09.723284 12193 hybrid_clock.cc:648] HybridClock initialized: now 1781851209723231 us; error 36 us; skew 500 ppm
I20260619 06:40:09.723582 12193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:09 dist-test-slave-qfk4 krb5kdc[11823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851209, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:09.726644 12193 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:40:09.727696 12193 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:09.727795 12193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:09.728091 12193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:09.731024 12193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7"
format_stamp: "Formatted at 2026-06-19 06:40:09 on dist-test-slave-qfk4"
server_key: "01f7e04034eb78bf80789dd37631c502"
server_key_iv: "0f775a249276b6c5cd60bb3737a97e61"
server_key_version: "encryptionkey@0"
I20260619 06:40:09.731808 12193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7"
format_stamp: "Formatted at 2026-06-19 06:40:09 on dist-test-slave-qfk4"
server_key: "01f7e04034eb78bf80789dd37631c502"
server_key_iv: "0f775a249276b6c5cd60bb3737a97e61"
server_key_version: "encryptionkey@0"
I20260619 06:40:09.736845 12193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 06:40:09.740339 12207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:09.741778 12193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:09.741914 12193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7"
format_stamp: "Formatted at 2026-06-19 06:40:09 on dist-test-slave-qfk4"
server_key: "01f7e04034eb78bf80789dd37631c502"
server_key_iv: "0f775a249276b6c5cd60bb3737a97e61"
server_key_version: "encryptionkey@0"
I20260619 06:40:09.742116 12193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:09.771101 12193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:09.775535 12193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:09.775770 12193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:09.776744 12193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:09.778195 12193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:09.778278 12193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:09.778352 12193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:09.778409 12193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:09.806957 12193 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:35895
I20260619 06:40:09.807011 12319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:35895 every 8 connection(s)
I20260619 06:40:09.807783 12193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:40:09.812269 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 12193
I20260619 06:40:09.812463 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:40:09.813012 7755 external_mini_cluster.cc:1428] Setting key 2bddca6a1ec15295aa52b7f95c1bef28
Jun 19 06:40:09 dist-test-slave-qfk4 krb5kdc[11823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851209, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:09.827173 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:09.810131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:37111 (local address 127.7.146.254:43781)
0619 06:40:09.810306 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:09.810310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:09.811524 (+ 1214us) server_negotiation.cc:407] Connection header received
0619 06:40:09.813803 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:09.813809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:09.813999 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:09.814090 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:09.815982 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.816744 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.818269 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.818685 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.818780 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:09.822029 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:09.822034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:09.822037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:09.822071 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:09.824327 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:09.825025 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:09.825029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:09.825032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:09.825154 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:09.825481 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:09.825484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:09.825486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:09.825738 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:09.825840 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:09.826629 (+ 789us) server_negotiation.cc:299] Negotiation successful
0619 06:40:09.826831 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:40:09.828256 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:09.810665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:43781 (local address 127.7.146.195:37111)
0619 06:40:09.811360 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:09.811401 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:09.813551 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:09.814370 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:09.814380 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:09.814972 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:09.815793 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:09.815821 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.816867 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.816871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:09.818133 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:09.818144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.818860 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.818865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:09.819072 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:09.819661 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:09.819687 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:09.821803 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:09.824459 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:09.824466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:09.824481 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:09.824921 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:09.825231 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:09.825233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:09.825236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:09.825392 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:09.825829 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:09.825835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:09.826201 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 06:40:09.826534 (+ 333us) client_negotiation.cc:240] Negotiation successful
0619 06:40:09.826996 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":296,"threads_started":1}
I20260619 06:40:09.829468 12320 heartbeater.cc:347] Connected to a master server at 127.7.146.254:43781
I20260619 06:40:09.829730 12320 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:09.830328 12320 heartbeater.cc:511] Master 127.7.146.254:43781 requested a full tablet report, sending...
I20260619 06:40:09.831647 11867 ts_manager.cc:194] Registered new tserver with Master: 0fb2ca9624104d3aa2d5a0a0fd85e9b7 (127.7.146.195:35895)
I20260619 06:40:09.832402 11867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:37111
I20260619 06:40:09.841147 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:40:09.857964 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:09.845722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.7.146.254:43781)
0619 06:40:09.845906 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:09.845912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:09.846471 (+ 559us) server_negotiation.cc:407] Connection header received
0619 06:40:09.846642 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:09.846647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:09.846727 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:09.846840 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:09.848540 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.849544 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.850766 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.851231 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.851342 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:09.853592 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:09.853597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:09.853600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:09.853634 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:09.855821 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:09.856439 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:09.856443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:09.856445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:09.856582 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:09.856873 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:09.856877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:09.856880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:09.857182 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:09.857356 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:09.857576 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 06:40:09.857731 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:40:09.862000 11867 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:40:09.868706 11867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:40:09.875226 12055 heartbeater.cc:503] Master 127.7.146.254:43781 was elected leader, sending a full tablet report...
I20260619 06:40:09.891969 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:09.880799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:38615 (local address 127.0.0.1:45492)
0619 06:40:09.882069 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:09.882109 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:09.882324 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:09.883342 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:09.883364 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:09.883406 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:09.883785 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:09.883797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.885889 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.885895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:09.888500 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:09.888518 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.890876 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.890883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:09.891119 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:09.891172 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 06:40:09.891313 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:40:09.891547 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":839,"threads_started":1}
I20260619 06:40:09.892724 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:09.880793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.7.146.194:38615)
0619 06:40:09.881498 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:09.881504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:09.882203 (+ 699us) server_negotiation.cc:407] Connection header received
0619 06:40:09.882580 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:09.882588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:09.882870 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:09.883062 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:09.883999 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.885719 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.888707 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.890333 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.890495 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:09.890725 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:09.892207 (+ 1482us) server_negotiation.cc:299] Negotiation successful
0619 06:40:09.892395 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":407,"threads_started":1}
I20260619 06:40:09.897293 12123 tablet_service.cc:1467] Processing CreateTablet for tablet e4d6ec58bfa9417995b7c114d6f127c0 (DEFAULT_TABLE table=test-table [id=6bda5cf693614e1a808242a655833264]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:09.898902 12123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d6ec58bfa9417995b7c114d6f127c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:09.900430 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:09.880811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:35895 (local address 127.0.0.1:34066)
0619 06:40:09.891776 (+ 10965us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:09.891800 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:09.891962 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:09.893330 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:09.893334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:09.893357 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:09.893663 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:09.893674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.895530 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.895535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:09.897550 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:09.897564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.899578 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.899584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:09.899760 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:09.899793 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:40:09.899880 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:40:09.900007 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10790,"mutex_wait_us":1,"thread_start_us":11257,"threads_started":1}
I20260619 06:40:09.901412 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:09.881334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.7.146.195:35895)
0619 06:40:09.881992 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:09.881999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:09.891906 (+ 9907us) server_negotiation.cc:407] Connection header received
0619 06:40:09.892127 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:09.892137 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:09.892411 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:09.892635 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:09.893860 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.895414 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.897727 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.899106 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.899221 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:09.899429 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:09.900941 (+ 1512us) server_negotiation.cc:299] Negotiation successful
0619 06:40:09.901104 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":350,"threads_started":1}
I20260619 06:40:09.902768 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:09.882495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:46697 (local address 127.0.0.1:35326)
0619 06:40:09.893067 (+ 10572us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:09.893087 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:09.893186 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:09.894417 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:09.894422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:09.894447 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:09.894645 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:09.894655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.896897 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.896903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:09.900282 (+ 3379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:09.900299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.902051 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.902055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:09.902178 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:09.902201 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:40:09.902278 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:40:09.902417 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10472,"mutex_wait_us":37,"thread_start_us":17945,"threads_started":1}
I20260619 06:40:09.902854 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:09.882867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.7.146.193:46697)
0619 06:40:09.893645 (+ 10778us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:09.893652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:09.893674 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:40:09.893752 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:09.893760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:09.894003 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:09.894203 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:09.894804 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.896751 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.900476 (+ 3725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:09.901657 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:09.901776 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:09.901975 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:09.902397 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 06:40:09.902552 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10635,"thread_start_us":10483,"threads_started":1}
I20260619 06:40:09.904903 12255 tablet_service.cc:1467] Processing CreateTablet for tablet e4d6ec58bfa9417995b7c114d6f127c0 (DEFAULT_TABLE table=test-table [id=6bda5cf693614e1a808242a655833264]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:09.906018 12255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d6ec58bfa9417995b7c114d6f127c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:09.905951 11990 tablet_service.cc:1467] Processing CreateTablet for tablet e4d6ec58bfa9417995b7c114d6f127c0 (DEFAULT_TABLE table=test-table [id=6bda5cf693614e1a808242a655833264]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:09.907058 11990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d6ec58bfa9417995b7c114d6f127c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:09.918094 12338 tablet_bootstrap.cc:492] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: Bootstrap starting.
I20260619 06:40:09.921160 12340 tablet_bootstrap.cc:492] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: Bootstrap starting.
I20260619 06:40:09.922878 12339 tablet_bootstrap.cc:492] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: Bootstrap starting.
I20260619 06:40:09.921797 12338 tablet_bootstrap.cc:654] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:09.924490 12338 log.cc:824] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:09.924791 12340 tablet_bootstrap.cc:654] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:09.925846 12340 log.cc:824] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:09.926076 12339 tablet_bootstrap.cc:654] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:09.927265 12339 log.cc:824] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:09.927661 12338 tablet_bootstrap.cc:492] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: No bootstrap required, opened a new log
I20260619 06:40:09.927937 12338 ts_tablet_manager.cc:1397] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 06:40:09.928726 12340 tablet_bootstrap.cc:492] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: No bootstrap required, opened a new log
I20260619 06:40:09.928985 12340 ts_tablet_manager.cc:1397] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260619 06:40:09.929687 12339 tablet_bootstrap.cc:492] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: No bootstrap required, opened a new log
I20260619 06:40:09.929899 12339 ts_tablet_manager.cc:1397] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:40:09.937005 12339 raft_consensus.cc:348] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9be44f44cf437cb252e3da4865b0d3" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38615 } } peers { permanent_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35895 } } peers { permanent_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46697 } }
I20260619 06:40:09.937390 12339 raft_consensus.cc:374] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:09.937480 12339 raft_consensus.cc:729] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb2ca9624104d3aa2d5a0a0fd85e9b7, State: Initialized, Role: FOLLOWER
I20260619 06:40:09.938064 12339 consensus_queue.cc:260] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9be44f44cf437cb252e3da4865b0d3" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38615 } } peers { permanent_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35895 } } peers { permanent_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46697 } }
I20260619 06:40:09.938297 12338 raft_consensus.cc:348] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9be44f44cf437cb252e3da4865b0d3" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38615 } } peers { permanent_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35895 } } peers { permanent_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46697 } }
I20260619 06:40:09.938720 12338 raft_consensus.cc:374] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:09.938818 12338 raft_consensus.cc:729] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c873a59d34344ba7807dafbbb6f8dcf2, State: Initialized, Role: FOLLOWER
I20260619 06:40:09.939224 12320 heartbeater.cc:503] Master 127.7.146.254:43781 was elected leader, sending a full tablet report...
I20260619 06:40:09.939625 12339 ts_tablet_manager.cc:1428] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 06:40:09.939502 12338 consensus_queue.cc:260] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9be44f44cf437cb252e3da4865b0d3" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38615 } } peers { permanent_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35895 } } peers { permanent_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46697 } }
I20260619 06:40:09.939807 12340 raft_consensus.cc:348] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9be44f44cf437cb252e3da4865b0d3" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38615 } } peers { permanent_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35895 } } peers { permanent_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46697 } }
I20260619 06:40:09.940268 12340 raft_consensus.cc:374] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:09.940393 12340 raft_consensus.cc:729] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c9be44f44cf437cb252e3da4865b0d3, State: Initialized, Role: FOLLOWER
I20260619 06:40:09.941164 12338 ts_tablet_manager.cc:1428] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260619 06:40:09.941057 12340 consensus_queue.cc:260] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9be44f44cf437cb252e3da4865b0d3" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38615 } } peers { permanent_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35895 } } peers { permanent_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46697 } }
I20260619 06:40:09.942852 12188 heartbeater.cc:503] Master 127.7.146.254:43781 was elected leader, sending a full tablet report...
I20260619 06:40:09.943552 12340 ts_tablet_manager.cc:1428] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
W20260619 06:40:10.059250 12321 tablet.cc:2369] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:40:10.075910 12189 tablet.cc:2369] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:40:10.097368 12056 tablet.cc:2369] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:40:10.136726 12346 raft_consensus.cc:482] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:40:10.137080 12346 raft_consensus.cc:504] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9be44f44cf437cb252e3da4865b0d3" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38615 } } peers { permanent_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35895 } } peers { permanent_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46697 } }
I20260619 06:40:10.139204 12346 leader_election.cc:288] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fb2ca9624104d3aa2d5a0a0fd85e9b7 (127.7.146.195:35895), c873a59d34344ba7807dafbbb6f8dcf2 (127.7.146.193:46697)
I20260619 06:40:10.146107 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.139596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:58651 (local address 127.7.146.195:35895)
0619 06:40:10.139780 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:10.139786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:10.140771 (+ 985us) server_negotiation.cc:407] Connection header received
0619 06:40:10.140936 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:10.140940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:10.140998 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:10.141075 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:10.141677 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.142385 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.144299 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.145174 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.145325 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:10.145395 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:10.145739 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 06:40:10.145882 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:40:10.146108 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.139832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:37489 (local address 127.7.146.193:46697)
0619 06:40:10.140004 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:10.140010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:10.140719 (+ 709us) server_negotiation.cc:407] Connection header received
0619 06:40:10.140899 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:10.140904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:10.140976 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:10.141048 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:10.141633 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.142379 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.144320 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.145122 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.145253 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:10.145328 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:10.145740 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 06:40:10.145865 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 06:40:10.146230 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.139886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:35895 (local address 127.7.146.194:58651)
0619 06:40:10.140661 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:10.140679 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:10.140816 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:10.141100 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:10.141104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:10.141144 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:10.141537 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:10.141546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.142519 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.142524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:10.144149 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:10.144165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.145392 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.145396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:10.145540 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:10.145596 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 06:40:10.145688 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:40:10.145819 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":21,"spinlock_wait_cycles":3328,"thread_start_us":400,"threads_started":1}
I20260619 06:40:10.146711 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.139583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:46697 (local address 127.7.146.194:37489)
0619 06:40:10.140605 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:10.140624 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:10.140763 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:10.141083 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:10.141088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:10.141118 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:10.141498 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:10.141511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.142546 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.142550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:10.144171 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:10.144183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.145341 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.145348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:10.145598 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:10.145622 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:40:10.145703 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 06:40:10.145830 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":74,"thread_start_us":354,"threads_started":1}
I20260619 06:40:10.147235 12275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d6ec58bfa9417995b7c114d6f127c0" candidate_uuid: "2c9be44f44cf437cb252e3da4865b0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" is_pre_election: true
I20260619 06:40:10.147563 12010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d6ec58bfa9417995b7c114d6f127c0" candidate_uuid: "2c9be44f44cf437cb252e3da4865b0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" is_pre_election: true
I20260619 06:40:10.147693 12275 raft_consensus.cc:2454] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c9be44f44cf437cb252e3da4865b0d3 in term 0.
I20260619 06:40:10.147994 12010 raft_consensus.cc:2454] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c9be44f44cf437cb252e3da4865b0d3 in term 0.
I20260619 06:40:10.148294 12078 leader_election.cc:302] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fb2ca9624104d3aa2d5a0a0fd85e9b7, 2c9be44f44cf437cb252e3da4865b0d3; no voters:
I20260619 06:40:10.148677 12346 raft_consensus.cc:2781] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:40:10.148826 12346 raft_consensus.cc:482] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:40:10.148927 12346 raft_consensus.cc:3037] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:10.150978 12346 raft_consensus.cc:504] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9be44f44cf437cb252e3da4865b0d3" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38615 } } peers { permanent_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35895 } } peers { permanent_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46697 } }
I20260619 06:40:10.151566 12346 leader_election.cc:288] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [CANDIDATE]: Term 1 election: Requested vote from peers 0fb2ca9624104d3aa2d5a0a0fd85e9b7 (127.7.146.195:35895), c873a59d34344ba7807dafbbb6f8dcf2 (127.7.146.193:46697)
I20260619 06:40:10.151965 12275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d6ec58bfa9417995b7c114d6f127c0" candidate_uuid: "2c9be44f44cf437cb252e3da4865b0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7"
I20260619 06:40:10.152004 12010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d6ec58bfa9417995b7c114d6f127c0" candidate_uuid: "2c9be44f44cf437cb252e3da4865b0d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c873a59d34344ba7807dafbbb6f8dcf2"
I20260619 06:40:10.152153 12275 raft_consensus.cc:3037] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:10.152168 12010 raft_consensus.cc:3037] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:10.154287 12010 raft_consensus.cc:2454] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c9be44f44cf437cb252e3da4865b0d3 in term 1.
I20260619 06:40:10.154371 12275 raft_consensus.cc:2454] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c9be44f44cf437cb252e3da4865b0d3 in term 1.
I20260619 06:40:10.154697 12077 leader_election.cc:302] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c9be44f44cf437cb252e3da4865b0d3, c873a59d34344ba7807dafbbb6f8dcf2; no voters:
I20260619 06:40:10.155032 12346 raft_consensus.cc:2781] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:10.155439 12346 raft_consensus.cc:686] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 1 LEADER]: Becoming Leader. State: Replica: 2c9be44f44cf437cb252e3da4865b0d3, State: Running, Role: LEADER
I20260619 06:40:10.155894 12346 consensus_queue.cc:237] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9be44f44cf437cb252e3da4865b0d3" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38615 } } peers { permanent_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35895 } } peers { permanent_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46697 } }
I20260619 06:40:10.160233 11866 catalog_manager.cc:5495] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c9be44f44cf437cb252e3da4865b0d3 (127.7.146.194). New cstate: current_term: 1 leader_uuid: "2c9be44f44cf437cb252e3da4865b0d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9be44f44cf437cb252e3da4865b0d3" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20260619 06:40:10.198376 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.193185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.7.146.194:38615)
0619 06:40:10.193382 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:10.193386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:10.193403 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:40:10.193556 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:10.193560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:10.193638 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:10.193758 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:40:10.194080 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.194780 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.196158 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.196549 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.196682 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:10.197168 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:40:10.197676 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:40:10.197783 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:10.198017 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 06:40:10.198094 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:40:10.206548 12275 raft_consensus.cc:1261] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7 [term 1 FOLLOWER]: Refusing update from remote peer 2c9be44f44cf437cb252e3da4865b0d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:10.206542 12010 raft_consensus.cc:1261] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2 [term 1 FOLLOWER]: Refusing update from remote peer 2c9be44f44cf437cb252e3da4865b0d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:10.207458 12346 consensus_queue.cc:1035] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb2ca9624104d3aa2d5a0a0fd85e9b7" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:10.207732 12349 consensus_queue.cc:1035] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c873a59d34344ba7807dafbbb6f8dcf2" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:10.232707 11866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:10.232969 11866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:10.236449 11866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbc9200892de4eb1ae719a64b034d422: Sending DeleteTablet for 3 replicas of tablet e4d6ec58bfa9417995b7c114d6f127c0
I20260619 06:40:10.237648 12123 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d6ec58bfa9417995b7c114d6f127c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:10 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:45492
I20260619 06:40:10.237830 11990 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d6ec58bfa9417995b7c114d6f127c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:10 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:35326
I20260619 06:40:10.238245 12255 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d6ec58bfa9417995b7c114d6f127c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:10 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:34066
I20260619 06:40:10.240279 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 11928
I20260619 06:40:10.266268 12364 tablet_replica.cc:330] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: stopping tablet replica
I20260619 06:40:10.266889 12364 raft_consensus.cc:2229] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:10.267508 12364 raft_consensus.cc:2258] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:10.268682 12362 tablet_replica.cc:330] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: stopping tablet replica
I20260619 06:40:10.269166 12362 raft_consensus.cc:2229] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:40:10.269898 12362 raft_consensus.cc:2258] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:10.272248 12363 tablet_replica.cc:330] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: stopping tablet replica
I20260619 06:40:10.272751 12364 ts_tablet_manager.cc:1905] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:10.272702 12363 raft_consensus.cc:2229] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:10.273243 12363 raft_consensus.cc:2258] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:10.276870 12362 ts_tablet_manager.cc:1905] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 06:40:10 dist-test-slave-qfk4 krb5kdc[11823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:40:10.280830 12364 ts_tablet_manager.cc:1918] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:10.281141 12364 log.cc:1196] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/e4d6ec58bfa9417995b7c114d6f127c0
I20260619 06:40:10.282027 12364 ts_tablet_manager.cc:1939] T e4d6ec58bfa9417995b7c114d6f127c0 P 0fb2ca9624104d3aa2d5a0a0fd85e9b7: Deleting consensus metadata
I20260619 06:40:10.282007 12363 ts_tablet_manager.cc:1905] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:10.283970 11853 catalog_manager.cc:4841] TS 0fb2ca9624104d3aa2d5a0a0fd85e9b7 (127.7.146.195:35895): tablet e4d6ec58bfa9417995b7c114d6f127c0 (table test-table [id=6bda5cf693614e1a808242a655833264]) successfully deleted
I20260619 06:40:10.286952 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.241580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.7.146.193:46697)
0619 06:40:10.241763 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:10.241769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:10.265410 (+ 23641us) server_negotiation.cc:407] Connection header received
0619 06:40:10.265673 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:10.265680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:10.265816 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:10.266050 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:10.270127 (+ 4077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.271334 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.273288 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.273997 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.274107 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:10.279530 (+ 5423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:10.279549 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:10.279569 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:10.279616 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:10.283602 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:10.284532 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:10.284541 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:10.284545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:10.284760 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:10.285188 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:10.285192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:10.285195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:10.285647 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:10.285999 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:10.286285 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 06:40:10.286549 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:40:10.287814 12050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:10.292547 12362 ts_tablet_manager.cc:1918] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:10.292766 12362 log.cc:1196] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/e4d6ec58bfa9417995b7c114d6f127c0
I20260619 06:40:10.293236 12362 ts_tablet_manager.cc:1939] T e4d6ec58bfa9417995b7c114d6f127c0 P 2c9be44f44cf437cb252e3da4865b0d3: Deleting consensus metadata
I20260619 06:40:10.294812 11854 catalog_manager.cc:4841] TS 2c9be44f44cf437cb252e3da4865b0d3 (127.7.146.194:38615): tablet e4d6ec58bfa9417995b7c114d6f127c0 (table test-table [id=6bda5cf693614e1a808242a655833264]) successfully deleted
I20260619 06:40:10.427930 12363 ts_tablet_manager.cc:1918] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:10.428166 12363 log.cc:1196] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/e4d6ec58bfa9417995b7c114d6f127c0
I20260619 06:40:10.428599 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 11928
I20260619 06:40:10.428714 12363 ts_tablet_manager.cc:1939] T e4d6ec58bfa9417995b7c114d6f127c0 P c873a59d34344ba7807dafbbb6f8dcf2: Deleting consensus metadata
I20260619 06:40:10.441772 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 12061
W20260619 06:40:10.441774 11852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.146.193:46697 (error 108)
I20260619 06:40:10.444067 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.443141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:46697 (local address 127.0.0.1:35340)
0619 06:40:10.443663 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:10.443817 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":319,"threads_started":1}
W20260619 06:40:10.444355 11852 catalog_manager.cc:4584] TS c873a59d34344ba7807dafbbb6f8dcf2 (127.7.146.193:46697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4d6ec58bfa9417995b7c114d6f127c0: Network error: Client connection negotiation failed: client connection to 127.7.146.193:46697: connect: Connection refused (error 111)
Jun 19 06:40:10 dist-test-slave-qfk4 krb5kdc[11823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:40:10.459120 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.442822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.7.146.194:38615)
0619 06:40:10.442977 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:10.442981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:10.442997 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:40:10.443050 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:10.443054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:10.443150 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:10.443247 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:10.444894 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.445738 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.447108 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.447556 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.447676 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:10.452147 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:10.452168 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:10.452194 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:10.452249 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:10.456042 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:10.457009 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:10.457020 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:10.457024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:10.457194 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:10.457554 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:10.457559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:10.457566 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:10.457998 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:10.458333 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:10.458493 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 06:40:10.458714 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:40:10.459805 12183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:10.498026 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.497477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:46697 (local address 127.0.0.1:35354)
0619 06:40:10.497727 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:10.497851 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 06:40:10.499104 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.498695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:46697 (local address 127.0.0.1:35370)
0619 06:40:10.498857 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:10.498943 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 06:40:10.559734 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.559174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:46697 (local address 127.0.0.1:35376)
0619 06:40:10.559430 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:10.559541 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 06:40:10.560861 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.560408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:46697 (local address 127.0.0.1:35382)
0619 06:40:10.560597 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:10.560686 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 06:40:10.572548 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 12061
I20260619 06:40:10.587523 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 12193
Jun 19 06:40:10 dist-test-slave-qfk4 krb5kdc[11823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:40:10.601679 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.588233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.7.146.195:35895)
0619 06:40:10.588386 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:10.588390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:10.588407 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:40:10.588512 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:10.588516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:10.588607 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:10.588696 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:10.589966 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.590676 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.591942 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:10.592349 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:10.592464 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:10.596430 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:10.596444 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:10.596460 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:10.596495 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:10.599011 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:10.599663 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:10.599670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:10.599672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:10.599812 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:10.600150 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:10.600154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:10.600156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:10.600521 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:10.600814 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:10.601153 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 06:40:10.601354 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:40:10.602319 12315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:10.641927 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.641384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:46697 (local address 127.0.0.1:35388)
0619 06:40:10.641633 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:10.641751 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 06:40:10.643069 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:10.642598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:46697 (local address 127.0.0.1:35400)
0619 06:40:10.642813 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:10.642904 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:46697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 06:40:10.690495 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 12193
I20260619 06:40:10.702903 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 11836
I20260619 06:40:10.704101 11897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:10.809441 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 11836
2026-06-19T06:40:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4164 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T06:40:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:10Z Disabled control of system clock
I20260619 06:40:10.870036 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:42231
--webserver_interface=127.7.146.254
--webserver_port=0
--builtin_ntp_servers=127.7.146.212:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:42231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:40:11.180811 12375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:11.181123 12375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:11.181252 12375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:11.188591 12375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:11.188712 12375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:11.188772 12375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:11.188849 12375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:11.188905 12375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:11.198488 12375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:42231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:42231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:11.200155 12375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:11.201943 12375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:11.213619 12380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:11.213568 12381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:11.213603 12383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:11.214339 12375 server_base.cc:1034] running on GCE node
I20260619 06:40:11.215152 12375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:11.216619 12375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:11.217890 12375 hybrid_clock.cc:648] HybridClock initialized: now 1781851211217828 us; error 54 us; skew 500 ppm
I20260619 06:40:11.218273 12375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:40:11.224678 12375 webserver.cc:460] Webserver started at http://127.7.146.254:39563/ using document root <none> and password file <none>
I20260619 06:40:11.225445 12375 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:11.225538 12375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:11.225838 12375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:11.228775 12375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "f355e8debf8e47e5b323513166f9d0da"
format_stamp: "Formatted at 2026-06-19 06:40:11 on dist-test-slave-qfk4"
server_key: "f9dbb740c01555772a1f963fddc14418"
server_key_iv: "b8247ada73f12cb7e45fbd39455d881b"
server_key_version: "encryptionkey@0"
I20260619 06:40:11.229631 12375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "f355e8debf8e47e5b323513166f9d0da"
format_stamp: "Formatted at 2026-06-19 06:40:11 on dist-test-slave-qfk4"
server_key: "f9dbb740c01555772a1f963fddc14418"
server_key_iv: "b8247ada73f12cb7e45fbd39455d881b"
server_key_version: "encryptionkey@0"
I20260619 06:40:11.234704 12375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:40:11.238052 12389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:11.239411 12375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:40:11.239574 12375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "f355e8debf8e47e5b323513166f9d0da"
format_stamp: "Formatted at 2026-06-19 06:40:11 on dist-test-slave-qfk4"
server_key: "f9dbb740c01555772a1f963fddc14418"
server_key_iv: "b8247ada73f12cb7e45fbd39455d881b"
server_key_version: "encryptionkey@0"
I20260619 06:40:11.239774 12375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:11.245622 12375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:11.246383 12375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:11.246654 12375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:11.263914 12375 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:42231
I20260619 06:40:11.263967 12440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:42231 every 8 connection(s)
I20260619 06:40:11.265290 12375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:11.266968 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 12375
I20260619 06:40:11.267139 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:11.267755 7755 external_mini_cluster.cc:1428] Setting key d3f19d6aea3f7f5d0035bc15f7eb6e32
I20260619 06:40:11.269316 12441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:11.277189 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:11.269764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.7.146.254:42231)
0619 06:40:11.270867 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:11.270893 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:11.270948 (+ 55us) server_negotiation.cc:407] Connection header received
0619 06:40:11.272750 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:11.272781 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:11.273413 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:11.273908 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:11.274992 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:11.275001 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:11.275019 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:11.275036 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:11.275130 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:11.275161 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:11.275379 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:11.275519 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 06:40:11.275823 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":347,"threads_started":1}
I20260619 06:40:11.282788 12441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da: Bootstrap starting.
I20260619 06:40:11.285710 12441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:11.286816 12441 log.cc:824] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:11.289563 12441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da: No bootstrap required, opened a new log
I20260619 06:40:11.296775 12441 raft_consensus.cc:348] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355e8debf8e47e5b323513166f9d0da" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42231 } }
I20260619 06:40:11.297236 12441 raft_consensus.cc:374] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:11.297333 12441 raft_consensus.cc:729] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f355e8debf8e47e5b323513166f9d0da, State: Initialized, Role: FOLLOWER
I20260619 06:40:11.297933 12441 consensus_queue.cc:260] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355e8debf8e47e5b323513166f9d0da" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42231 } }
I20260619 06:40:11.298146 12441 raft_consensus.cc:388] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:11.298270 12441 raft_consensus.cc:482] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:11.298432 12441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:11.300474 12441 raft_consensus.cc:504] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355e8debf8e47e5b323513166f9d0da" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42231 } }
I20260619 06:40:11.301059 12441 leader_election.cc:302] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f355e8debf8e47e5b323513166f9d0da; no voters:
I20260619 06:40:11.301640 12441 leader_election.cc:288] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:11.301769 12446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:11.302488 12446 raft_consensus.cc:686] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [term 1 LEADER]: Becoming Leader. State: Replica: f355e8debf8e47e5b323513166f9d0da, State: Running, Role: LEADER
I20260619 06:40:11.303020 12446 consensus_queue.cc:237] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355e8debf8e47e5b323513166f9d0da" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42231 } }
I20260619 06:40:11.303655 12441 sys_catalog.cc:564] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:11.309664 12447 sys_catalog.cc:455] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f355e8debf8e47e5b323513166f9d0da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355e8debf8e47e5b323513166f9d0da" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42231 } } }
I20260619 06:40:11.310043 12447 sys_catalog.cc:458] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:11.309743 12448 sys_catalog.cc:455] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [sys.catalog]: SysCatalogTable state changed. Reason: New leader f355e8debf8e47e5b323513166f9d0da. Latest consensus state: current_term: 1 leader_uuid: "f355e8debf8e47e5b323513166f9d0da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355e8debf8e47e5b323513166f9d0da" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42231 } } }
I20260619 06:40:11.310173 12448 sys_catalog.cc:458] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da [sys.catalog]: This master's current role is: LEADER
W20260619 06:40:11.315380 12460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:40:11.315505 12460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:40:11.315636 12461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:11.317739 12461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:11.326545 12461 catalog_manager.cc:1347] Generated new cluster ID: 8ee08da20835459d82989cd7c8c49b75
I20260619 06:40:11.326661 12461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:11.382836 12461 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:11.383360 12461 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:11.391333 12461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da: Generated new TSK 0
I20260619 06:40:11.392347 12461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:40:11.410329 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:42231
--builtin_ntp_servers=127.7.146.212:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:40:11.727025 12465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:11.727326 12465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:11.727463 12465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:11.734740 12465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:11.734843 12465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:11.734994 12465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:40:11.744393 12465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:42231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:11.745695 12465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:11.747506 12465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:11.761600 12470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:11.761410 12471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:11.761519 12473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:11.762558 12465 server_base.cc:1034] running on GCE node
I20260619 06:40:11.763195 12465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:11.764317 12465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:11.765545 12465 hybrid_clock.cc:648] HybridClock initialized: now 1781851211765483 us; error 55 us; skew 500 ppm
I20260619 06:40:11.765848 12465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:40:11.768160 12465 webserver.cc:460] Webserver started at http://127.7.146.193:36489/ using document root <none> and password file <none>
I20260619 06:40:11.768918 12465 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:11.769013 12465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:11.769280 12465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:11.772168 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "74bcefa0f2744317b352f797378c0001"
format_stamp: "Formatted at 2026-06-19 06:40:11 on dist-test-slave-qfk4"
server_key: "08ef94b1b9c437a8244119dd48c0211e"
server_key_iv: "ef576150b96026843abc2950b251bd12"
server_key_version: "encryptionkey@0"
I20260619 06:40:11.773028 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "74bcefa0f2744317b352f797378c0001"
format_stamp: "Formatted at 2026-06-19 06:40:11 on dist-test-slave-qfk4"
server_key: "08ef94b1b9c437a8244119dd48c0211e"
server_key_iv: "ef576150b96026843abc2950b251bd12"
server_key_version: "encryptionkey@0"
I20260619 06:40:11.778321 12465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:40:11.782243 12479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:11.783757 12465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:40:11.783952 12465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "74bcefa0f2744317b352f797378c0001"
format_stamp: "Formatted at 2026-06-19 06:40:11 on dist-test-slave-qfk4"
server_key: "08ef94b1b9c437a8244119dd48c0211e"
server_key_iv: "ef576150b96026843abc2950b251bd12"
server_key_version: "encryptionkey@0"
I20260619 06:40:11.784188 12465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:11.789801 12465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:11.790607 12465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:11.790856 12465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:11.791801 12465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:11.793226 12465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:11.793313 12465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:11.793431 12465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:11.793501 12465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:11.822811 12465 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:42677
I20260619 06:40:11.822857 12591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:42677 every 8 connection(s)
I20260619 06:40:11.824047 12465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:40:11.827746 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 12465
I20260619 06:40:11.827906 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:40:11.828583 7755 external_mini_cluster.cc:1428] Setting key 22c5be9b93ee1d820e6b33f762ea0b34
I20260619 06:40:11.831831 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:42231
--builtin_ntp_servers=127.7.146.212:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 06:40:11.832940 12594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:11.826476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:33859 (local address 127.7.146.254:42231)
0619 06:40:11.826886 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:11.826890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:11.828367 (+ 1477us) server_negotiation.cc:407] Connection header received
0619 06:40:11.830330 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:11.830335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:11.830393 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:11.830503 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:11.832293 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:11.832297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:11.832300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:11.832316 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:11.832357 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:11.832369 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:11.832461 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:11.832680 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 06:40:11.832728 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":202,"threads_started":1}
I20260619 06:40:11.834225 12595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:11.827114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:42231 (local address 127.7.146.193:33859)
0619 06:40:11.828190 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:11.828233 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:11.830036 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:11.830756 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:11.830767 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:11.831017 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:11.831952 (+ 935us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:40:11.831956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:11.832019 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:40:11.832131 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:11.832458 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:11.832466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:11.832473 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:40:11.832765 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 06:40:11.833037 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":336,"threads_started":1}
I20260619 06:40:11.835642 12592 heartbeater.cc:347] Connected to a master server at 127.7.146.254:42231
I20260619 06:40:11.835942 12592 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:11.836692 12592 heartbeater.cc:511] Master 127.7.146.254:42231 requested a full tablet report, sending...
I20260619 06:40:11.838894 12406 ts_manager.cc:194] Registered new tserver with Master: 74bcefa0f2744317b352f797378c0001 (127.7.146.193:42677)
W20260619 06:40:12.141392 12596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:12.141681 12596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:12.141768 12596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:12.150177 12596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:12.150287 12596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:12.150400 12596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:40:12.159356 12596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:42231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:12.160595 12596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:12.162352 12596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:12.177459 12604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:12.177394 12601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:12.177438 12602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:12.178225 12596 server_base.cc:1034] running on GCE node
I20260619 06:40:12.178880 12596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:12.179813 12596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:12.181026 12596 hybrid_clock.cc:648] HybridClock initialized: now 1781851212180971 us; error 47 us; skew 500 ppm
I20260619 06:40:12.181339 12596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:40:12.183454 12596 webserver.cc:460] Webserver started at http://127.7.146.194:33431/ using document root <none> and password file <none>
I20260619 06:40:12.184109 12596 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:12.184204 12596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:12.184502 12596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:12.187404 12596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "77e315cc324344468ac23d4fad7e40a9"
format_stamp: "Formatted at 2026-06-19 06:40:12 on dist-test-slave-qfk4"
server_key: "ff9b3eca28c9067c1c9d14bafd068694"
server_key_iv: "1251f3d9675dd943d867a48a21ce6cbc"
server_key_version: "encryptionkey@0"
I20260619 06:40:12.188191 12596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "77e315cc324344468ac23d4fad7e40a9"
format_stamp: "Formatted at 2026-06-19 06:40:12 on dist-test-slave-qfk4"
server_key: "ff9b3eca28c9067c1c9d14bafd068694"
server_key_iv: "1251f3d9675dd943d867a48a21ce6cbc"
server_key_version: "encryptionkey@0"
I20260619 06:40:12.193670 12596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:40:12.197865 12610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:12.199461 12596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:40:12.199632 12596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "77e315cc324344468ac23d4fad7e40a9"
format_stamp: "Formatted at 2026-06-19 06:40:12 on dist-test-slave-qfk4"
server_key: "ff9b3eca28c9067c1c9d14bafd068694"
server_key_iv: "1251f3d9675dd943d867a48a21ce6cbc"
server_key_version: "encryptionkey@0"
I20260619 06:40:12.199859 12596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:12.205641 12596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:12.206408 12596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:12.206646 12596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:12.207520 12596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:12.208925 12596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:12.209019 12596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:12.209120 12596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:12.209185 12596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:12.237862 12596 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:32879
I20260619 06:40:12.237916 12722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:32879 every 8 connection(s)
I20260619 06:40:12.239141 12596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:40:12.241267 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 12596
I20260619 06:40:12.241413 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:40:12.241943 7755 external_mini_cluster.cc:1428] Setting key d5b114e002e32c5636b73e90d72cacbe
I20260619 06:40:12.245062 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:42231
--builtin_ntp_servers=127.7.146.212:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 06:40:12.247980 12594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.241707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:42053 (local address 127.7.146.254:42231)
0619 06:40:12.241930 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:12.241936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:12.243569 (+ 1633us) server_negotiation.cc:407] Connection header received
0619 06:40:12.245813 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:12.245817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:12.245866 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:12.245945 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:12.247327 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:12.247332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:12.247337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:12.247354 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:12.247387 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:12.247401 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:12.247531 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:12.247673 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 06:40:12.247726 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:40:12.249207 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.242302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:42231 (local address 127.7.146.194:42053)
0619 06:40:12.243423 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:12.243473 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:12.245530 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:12.246218 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:12.246230 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:12.246359 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:12.247006 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:40:12.247009 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:12.247057 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:40:12.247162 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:12.247486 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:12.247494 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:12.247500 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:40:12.247746 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 06:40:12.248016 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":332,"threads_started":1}
I20260619 06:40:12.250566 12723 heartbeater.cc:347] Connected to a master server at 127.7.146.254:42231
I20260619 06:40:12.250912 12723 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:12.251503 12723 heartbeater.cc:511] Master 127.7.146.254:42231 requested a full tablet report, sending...
I20260619 06:40:12.252863 12406 ts_manager.cc:194] Registered new tserver with Master: 77e315cc324344468ac23d4fad7e40a9 (127.7.146.194:32879)
W20260619 06:40:12.563426 12726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:12.563735 12726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:12.563874 12726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:12.571334 12726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:12.571446 12726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:12.571593 12726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:40:12.581041 12726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:42231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:12.582337 12726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:12.584239 12726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:12.596840 12731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:12.596805 12734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:12.596877 12732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:12.598310 12726 server_base.cc:1034] running on GCE node
I20260619 06:40:12.598938 12726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:12.599960 12726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:12.601174 12726 hybrid_clock.cc:648] HybridClock initialized: now 1781851212601128 us; error 35 us; skew 500 ppm
I20260619 06:40:12.601480 12726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:40:12.603832 12726 webserver.cc:460] Webserver started at http://127.7.146.195:35505/ using document root <none> and password file <none>
I20260619 06:40:12.604544 12726 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:12.604686 12726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:12.604993 12726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:12.607880 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "5bf19f7723ab42b8bda3139abeb5dc6d"
format_stamp: "Formatted at 2026-06-19 06:40:12 on dist-test-slave-qfk4"
server_key: "02d92c0215688bf8dac6a87769337937"
server_key_iv: "d7e4882cdc8faf1880f4636651551530"
server_key_version: "encryptionkey@0"
I20260619 06:40:12.608659 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "5bf19f7723ab42b8bda3139abeb5dc6d"
format_stamp: "Formatted at 2026-06-19 06:40:12 on dist-test-slave-qfk4"
server_key: "02d92c0215688bf8dac6a87769337937"
server_key_iv: "d7e4882cdc8faf1880f4636651551530"
server_key_version: "encryptionkey@0"
I20260619 06:40:12.613705 12726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:40:12.617098 12740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:12.618413 12726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:40:12.618563 12726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "5bf19f7723ab42b8bda3139abeb5dc6d"
format_stamp: "Formatted at 2026-06-19 06:40:12 on dist-test-slave-qfk4"
server_key: "02d92c0215688bf8dac6a87769337937"
server_key_iv: "d7e4882cdc8faf1880f4636651551530"
server_key_version: "encryptionkey@0"
I20260619 06:40:12.618777 12726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:12.624677 12726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:12.625509 12726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:12.625880 12726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:12.626796 12726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:12.628113 12726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:12.628204 12726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:12.628310 12726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:12.628377 12726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:12.657557 12726 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:40993
I20260619 06:40:12.657608 12852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:40993 every 8 connection(s)
I20260619 06:40:12.658834 12726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:40:12.665139 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 12726
I20260619 06:40:12.665294 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:40:12.665844 7755 external_mini_cluster.cc:1428] Setting key 28f306283f42a1d2f0ec825d4319531d
I20260619 06:40:12.666812 12594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.661352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:60083 (local address 127.7.146.254:42231)
0619 06:40:12.661546 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:12.661550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:12.662885 (+ 1335us) server_negotiation.cc:407] Connection header received
0619 06:40:12.664554 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:12.664589 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:12.664658 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:12.664744 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:12.666096 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:12.666102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:12.666107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:12.666125 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:12.666156 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:12.666168 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:12.666270 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:12.666448 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 06:40:12.666529 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:40:12.667877 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.661850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:42231 (local address 127.7.146.195:60083)
0619 06:40:12.662734 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:12.662768 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:12.664282 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:12.664979 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:12.664986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:12.665091 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:12.665770 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:40:12.665772 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:12.665809 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:40:12.665897 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:12.666250 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:12.666260 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:12.666267 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:40:12.666515 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 06:40:12.666770 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":250,"threads_started":1}
I20260619 06:40:12.669153 12853 heartbeater.cc:347] Connected to a master server at 127.7.146.254:42231
I20260619 06:40:12.669449 12853 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:12.670063 12853 heartbeater.cc:511] Master 127.7.146.254:42231 requested a full tablet report, sending...
I20260619 06:40:12.671387 12406 ts_manager.cc:194] Registered new tserver with Master: 5bf19f7723ab42b8bda3139abeb5dc6d (127.7.146.195:40993)
I20260619 06:40:12.680965 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:40:12.687719 12594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.685364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.7.146.254:42231)
0619 06:40:12.685608 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:12.685614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:12.685893 (+ 279us) server_negotiation.cc:407] Connection header received
0619 06:40:12.686054 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:12.686057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:12.686111 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:12.686190 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:12.687126 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:12.687131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:12.687134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:12.687149 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:12.687180 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:12.687191 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:12.687283 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:12.687378 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 06:40:12.687425 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:40:12.691774 12406 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:40:12.698400 12406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:40:12.722554 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.710920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.7.146.193:42677)
0619 06:40:12.719458 (+ 8538us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:12.719465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:12.719488 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:40:12.719545 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:12.719550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:12.719753 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:12.719898 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:12.721287 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:12.721296 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:12.721331 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:12.721351 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:12.721415 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:12.721441 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:12.721567 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:12.722195 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 06:40:12.722297 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8415,"thread_start_us":8326,"threads_started":1}
I20260619 06:40:12.722960 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.710416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:42677 (local address 127.0.0.1:48318)
0619 06:40:12.717205 (+ 6789us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:12.717263 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:12.717423 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:12.720037 (+ 2614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:12.720045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:12.720144 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:12.720959 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:40:12.720962 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:12.720991 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:40:12.721085 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:12.721621 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:12.721631 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:12.721640 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:40:12.721867 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 06:40:12.721957 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6482,"thread_start_us":6303,"threads_started":1}
I20260619 06:40:12.724694 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.711074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.7.146.195:40993)
0619 06:40:12.719113 (+ 8039us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:12.719120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:12.719970 (+ 850us) server_negotiation.cc:407] Connection header received
0619 06:40:12.720223 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:12.720233 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:12.720469 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:12.720653 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:12.723651 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:12.723659 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:12.723673 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:12.723693 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:12.723759 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:12.723790 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:12.723964 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:12.724181 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 06:40:12.724283 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7887,"thread_start_us":7747,"threads_started":1}
I20260619 06:40:12.725438 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.710679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:40993 (local address 127.0.0.1:41444)
0619 06:40:12.719853 (+ 9174us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:12.719871 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:12.720058 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:12.720943 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:12.720947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:12.720970 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:12.723453 (+ 2483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:40:12.723456 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:12.723472 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:40:12.723513 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:12.724022 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:12.724027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:12.724032 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:40:12.724996 (+ 964us) client_negotiation.cc:240] Negotiation successful
0619 06:40:12.725045 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9039,"thread_start_us":8918,"threads_started":1}
I20260619 06:40:12.726262 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.722093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:32879 (local address 127.0.0.1:52510)
0619 06:40:12.723178 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:12.723194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:12.723283 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:12.724901 (+ 1618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:12.724905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:12.724948 (+ 43us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:12.725481 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:40:12.725484 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:12.725501 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:40:12.725545 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:12.725902 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:12.725906 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:12.725911 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:40:12.725992 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 06:40:12.726039 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"threads_started":1}
I20260619 06:40:12.726678 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.720330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.7.146.194:32879)
0619 06:40:12.722077 (+ 1747us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:12.722085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:12.723290 (+ 1205us) server_negotiation.cc:407] Connection header received
0619 06:40:12.723380 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:12.723403 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:12.723633 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:12.723853 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:12.725703 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:12.725710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:12.725720 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:12.725742 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:12.725784 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:12.725808 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:12.725952 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:12.726018 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 06:40:12.726107 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1552,"thread_start_us":1432,"threads_started":1}
I20260619 06:40:12.727475 12527 tablet_service.cc:1467] Processing CreateTablet for tablet 30ad9ac7a90047a780832b99dca3b152 (DEFAULT_TABLE table=test-table [id=707b180a31a24414892803aa12f647ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:12.729130 12527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ad9ac7a90047a780832b99dca3b152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:12.732039 12788 tablet_service.cc:1467] Processing CreateTablet for tablet 30ad9ac7a90047a780832b99dca3b152 (DEFAULT_TABLE table=test-table [id=707b180a31a24414892803aa12f647ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:12.732036 12658 tablet_service.cc:1467] Processing CreateTablet for tablet 30ad9ac7a90047a780832b99dca3b152 (DEFAULT_TABLE table=test-table [id=707b180a31a24414892803aa12f647ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:12.733461 12788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ad9ac7a90047a780832b99dca3b152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:12.733480 12658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ad9ac7a90047a780832b99dca3b152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:12.744342 12871 tablet_bootstrap.cc:492] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9: Bootstrap starting.
I20260619 06:40:12.746623 12872 tablet_bootstrap.cc:492] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: Bootstrap starting.
I20260619 06:40:12.747530 12873 tablet_bootstrap.cc:492] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: Bootstrap starting.
I20260619 06:40:12.748165 12871 tablet_bootstrap.cc:654] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:12.749845 12871 log.cc:824] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:12.750674 12872 tablet_bootstrap.cc:654] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:12.751140 12873 tablet_bootstrap.cc:654] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:12.751863 12872 log.cc:824] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:12.752080 12873 log.cc:824] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:12.753046 12871 tablet_bootstrap.cc:492] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9: No bootstrap required, opened a new log
I20260619 06:40:12.753334 12871 ts_tablet_manager.cc:1397] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 06:40:12.754880 12872 tablet_bootstrap.cc:492] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: No bootstrap required, opened a new log
I20260619 06:40:12.755131 12872 ts_tablet_manager.cc:1397] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 06:40:12.756770 12873 tablet_bootstrap.cc:492] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: No bootstrap required, opened a new log
I20260619 06:40:12.757050 12873 ts_tablet_manager.cc:1397] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260619 06:40:12.763566 12871 raft_consensus.cc:348] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bcefa0f2744317b352f797378c0001" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42677 } } peers { permanent_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 40993 } } peers { permanent_uuid: "77e315cc324344468ac23d4fad7e40a9" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 32879 } }
I20260619 06:40:12.764060 12871 raft_consensus.cc:374] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:12.764192 12871 raft_consensus.cc:729] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e315cc324344468ac23d4fad7e40a9, State: Initialized, Role: FOLLOWER
I20260619 06:40:12.764014 12873 raft_consensus.cc:348] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bcefa0f2744317b352f797378c0001" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42677 } } peers { permanent_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 40993 } } peers { permanent_uuid: "77e315cc324344468ac23d4fad7e40a9" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 32879 } }
I20260619 06:40:12.764420 12873 raft_consensus.cc:374] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:12.764552 12873 raft_consensus.cc:729] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf19f7723ab42b8bda3139abeb5dc6d, State: Initialized, Role: FOLLOWER
I20260619 06:40:12.764881 12871 consensus_queue.cc:260] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bcefa0f2744317b352f797378c0001" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42677 } } peers { permanent_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 40993 } } peers { permanent_uuid: "77e315cc324344468ac23d4fad7e40a9" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 32879 } }
I20260619 06:40:12.765110 12873 consensus_queue.cc:260] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bcefa0f2744317b352f797378c0001" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42677 } } peers { permanent_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 40993 } } peers { permanent_uuid: "77e315cc324344468ac23d4fad7e40a9" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 32879 } }
I20260619 06:40:12.765127 12872 raft_consensus.cc:348] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bcefa0f2744317b352f797378c0001" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42677 } } peers { permanent_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 40993 } } peers { permanent_uuid: "77e315cc324344468ac23d4fad7e40a9" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 32879 } }
I20260619 06:40:12.765543 12872 raft_consensus.cc:374] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:12.765697 12872 raft_consensus.cc:729] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bcefa0f2744317b352f797378c0001, State: Initialized, Role: FOLLOWER
I20260619 06:40:12.766155 12853 heartbeater.cc:503] Master 127.7.146.254:42231 was elected leader, sending a full tablet report...
I20260619 06:40:12.766467 12872 consensus_queue.cc:260] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bcefa0f2744317b352f797378c0001" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42677 } } peers { permanent_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 40993 } } peers { permanent_uuid: "77e315cc324344468ac23d4fad7e40a9" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 32879 } }
I20260619 06:40:12.766829 12873 ts_tablet_manager.cc:1428] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260619 06:40:12.769275 12723 heartbeater.cc:503] Master 127.7.146.254:42231 was elected leader, sending a full tablet report...
I20260619 06:40:12.769999 12871 ts_tablet_manager.cc:1428] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260619 06:40:12.770833 12592 heartbeater.cc:503] Master 127.7.146.254:42231 was elected leader, sending a full tablet report...
I20260619 06:40:12.772079 12872 ts_tablet_manager.cc:1428] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
W20260619 06:40:12.826200 12593 tablet.cc:2369] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:40:12.893772 12877 raft_consensus.cc:482] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:40:12.894117 12877 raft_consensus.cc:504] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bcefa0f2744317b352f797378c0001" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42677 } } peers { permanent_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 40993 } } peers { permanent_uuid: "77e315cc324344468ac23d4fad7e40a9" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 32879 } }
I20260619 06:40:12.895859 12877 leader_election.cc:288] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74bcefa0f2744317b352f797378c0001 (127.7.146.193:42677), 5bf19f7723ab42b8bda3139abeb5dc6d (127.7.146.195:40993)
I20260619 06:40:12.899320 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.896415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:55729 (local address 127.7.146.195:40993)
0619 06:40:12.896611 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:12.896616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:12.897301 (+ 685us) server_negotiation.cc:407] Connection header received
0619 06:40:12.897469 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:12.897474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:12.897548 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:12.897646 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:12.898735 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:12.898741 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:12.898745 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:12.898765 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:12.898800 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:12.898815 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:12.898934 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:12.899018 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 06:40:12.899077 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:40:12.899351 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.896404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:34355 (local address 127.7.146.193:42677)
0619 06:40:12.896605 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:12.896611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:12.897320 (+ 709us) server_negotiation.cc:407] Connection header received
0619 06:40:12.897468 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:12.897474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:12.897538 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:12.897646 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:12.898735 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:12.898740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:12.898745 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:12.898765 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:12.898800 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:12.898815 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:12.898929 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:12.899038 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 06:40:12.899089 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:40:12.899569 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.896163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:40993 (local address 127.7.146.194:55729)
0619 06:40:12.897172 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:12.897191 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:12.897351 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:12.897680 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:12.897684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:12.897709 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:12.898509 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:40:12.898513 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:12.898534 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:40:12.898583 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:12.898927 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:12.898930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:12.898934 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:40:12.899051 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 06:40:12.899083 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"spinlock_wait_cycles":896,"thread_start_us":690,"threads_started":1}
I20260619 06:40:12.899715 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.896218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:42677 (local address 127.7.146.194:34355)
0619 06:40:12.897230 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:12.897243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:12.897359 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:12.897680 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:12.897684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:12.897709 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:12.898567 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:40:12.898570 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:12.898581 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:40:12.898608 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:12.898911 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:12.898916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:12.898921 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:40:12.899023 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 06:40:12.899080 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"spinlock_wait_cycles":13952,"thread_start_us":589,"threads_started":1}
I20260619 06:40:12.900693 12547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad9ac7a90047a780832b99dca3b152" candidate_uuid: "77e315cc324344468ac23d4fad7e40a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bcefa0f2744317b352f797378c0001" is_pre_election: true
I20260619 06:40:12.900805 12808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad9ac7a90047a780832b99dca3b152" candidate_uuid: "77e315cc324344468ac23d4fad7e40a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" is_pre_election: true
I20260619 06:40:12.901289 12808 raft_consensus.cc:2454] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e315cc324344468ac23d4fad7e40a9 in term 0.
I20260619 06:40:12.901292 12547 raft_consensus.cc:2454] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e315cc324344468ac23d4fad7e40a9 in term 0.
I20260619 06:40:12.901952 12613 leader_election.cc:302] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf19f7723ab42b8bda3139abeb5dc6d, 77e315cc324344468ac23d4fad7e40a9; no voters:
I20260619 06:40:12.902329 12877 raft_consensus.cc:2781] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:40:12.902468 12877 raft_consensus.cc:482] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:40:12.902563 12877 raft_consensus.cc:3037] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:12.904882 12877 raft_consensus.cc:504] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bcefa0f2744317b352f797378c0001" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42677 } } peers { permanent_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 40993 } } peers { permanent_uuid: "77e315cc324344468ac23d4fad7e40a9" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 32879 } }
I20260619 06:40:12.905413 12877 leader_election.cc:288] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [CANDIDATE]: Term 1 election: Requested vote from peers 74bcefa0f2744317b352f797378c0001 (127.7.146.193:42677), 5bf19f7723ab42b8bda3139abeb5dc6d (127.7.146.195:40993)
I20260619 06:40:12.905973 12547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad9ac7a90047a780832b99dca3b152" candidate_uuid: "77e315cc324344468ac23d4fad7e40a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bcefa0f2744317b352f797378c0001"
I20260619 06:40:12.906121 12808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ad9ac7a90047a780832b99dca3b152" candidate_uuid: "77e315cc324344468ac23d4fad7e40a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d"
I20260619 06:40:12.906193 12547 raft_consensus.cc:3037] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:12.906322 12808 raft_consensus.cc:3037] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:12.909365 12808 raft_consensus.cc:2454] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e315cc324344468ac23d4fad7e40a9 in term 1.
I20260619 06:40:12.909368 12547 raft_consensus.cc:2454] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e315cc324344468ac23d4fad7e40a9 in term 1.
I20260619 06:40:12.909806 12612 leader_election.cc:302] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74bcefa0f2744317b352f797378c0001, 77e315cc324344468ac23d4fad7e40a9; no voters:
I20260619 06:40:12.910178 12877 raft_consensus.cc:2781] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 1 FOLLOWER]: Leader election won for term 1
W20260619 06:40:12.910317 12854 tablet.cc:2369] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:40:12.911357 12877 raft_consensus.cc:686] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 1 LEADER]: Becoming Leader. State: Replica: 77e315cc324344468ac23d4fad7e40a9, State: Running, Role: LEADER
I20260619 06:40:12.911816 12877 consensus_queue.cc:237] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bcefa0f2744317b352f797378c0001" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42677 } } peers { permanent_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 40993 } } peers { permanent_uuid: "77e315cc324344468ac23d4fad7e40a9" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 32879 } }
I20260619 06:40:12.916487 12405 catalog_manager.cc:5495] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e315cc324344468ac23d4fad7e40a9 (127.7.146.194). New cstate: current_term: 1 leader_uuid: "77e315cc324344468ac23d4fad7e40a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bcefa0f2744317b352f797378c0001" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 40993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e315cc324344468ac23d4fad7e40a9" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 32879 } health_report { overall_health: HEALTHY } } }
I20260619 06:40:12.955266 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.953386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.7.146.194:32879)
0619 06:40:12.953558 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:12.953562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:12.953581 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:40:12.953734 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:12.953737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:12.953817 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:12.953899 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:12.954699 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:12.954703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:12.954706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:12.954721 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:12.954751 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:12.954763 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:12.954854 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:12.954962 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 06:40:12.955022 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:40:12.963274 12808 raft_consensus.cc:1261] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d [term 1 FOLLOWER]: Refusing update from remote peer 77e315cc324344468ac23d4fad7e40a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:12.963274 12547 raft_consensus.cc:1261] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001 [term 1 FOLLOWER]: Refusing update from remote peer 77e315cc324344468ac23d4fad7e40a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:12.964257 12882 consensus_queue.cc:1035] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74bcefa0f2744317b352f797378c0001" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:12.964601 12877 consensus_queue.cc:1035] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf19f7723ab42b8bda3139abeb5dc6d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 40993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:12.979364 12887 mvcc.cc:205] Tried to move back new op lower bound from 7298462568287854592 to 7298462568090640384. Current Snapshot: MvccSnapshot[applied={T|T < 7298462568287854592}]
I20260619 06:40:12.988044 12405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:12.988349 12405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:12.992045 12405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f355e8debf8e47e5b323513166f9d0da: Sending DeleteTablet for 3 replicas of tablet 30ad9ac7a90047a780832b99dca3b152
I20260619 06:40:12.993275 12527 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ad9ac7a90047a780832b99dca3b152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:12 UTC) from {username='slave'} at 127.0.0.1:48318
I20260619 06:40:12.993312 12788 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ad9ac7a90047a780832b99dca3b152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:12 UTC) from {username='slave'} at 127.0.0.1:41444
I20260619 06:40:12.993929 12658 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ad9ac7a90047a780832b99dca3b152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:12 UTC) from {username='slave'} at 127.0.0.1:52510
I20260619 06:40:12.994092 12895 tablet_replica.cc:330] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: stopping tablet replica
I20260619 06:40:12.994552 12895 raft_consensus.cc:2229] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:12.995081 12895 raft_consensus.cc:2258] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:12.995981 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 12465
I20260619 06:40:13.003909 12897 tablet_replica.cc:330] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9: stopping tablet replica
I20260619 06:40:13.004621 12897 raft_consensus.cc:2229] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:40:13.005465 12897 raft_consensus.cc:2258] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:13.006448 12895 ts_tablet_manager.cc:1905] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:13.006829 12896 tablet_replica.cc:330] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: stopping tablet replica
I20260619 06:40:13.007586 12896 raft_consensus.cc:2229] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:13.007870 12897 ts_tablet_manager.cc:1905] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:13.009285 12896 raft_consensus.cc:2258] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:13.011092 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:12.998238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.7.146.193:42677)
0619 06:40:12.998410 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:12.998418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:13.006551 (+ 8133us) server_negotiation.cc:407] Connection header received
0619 06:40:13.007848 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:13.007854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:13.007955 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:13.008053 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:13.009943 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:13.009949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:13.009954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:13.009976 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:13.010029 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:13.010046 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:13.010143 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:13.010698 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 06:40:13.010766 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:40:13.013692 12896 ts_tablet_manager.cc:1905] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:13.014190 12895 ts_tablet_manager.cc:1918] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:13.014424 12895 log.cc:1196] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/30ad9ac7a90047a780832b99dca3b152
I20260619 06:40:13.015002 12897 ts_tablet_manager.cc:1918] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:13.015034 12895 ts_tablet_manager.cc:1939] T 30ad9ac7a90047a780832b99dca3b152 P 74bcefa0f2744317b352f797378c0001: Deleting consensus metadata
I20260619 06:40:13.015174 12897 log.cc:1196] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/30ad9ac7a90047a780832b99dca3b152
I20260619 06:40:13.015688 12897 ts_tablet_manager.cc:1939] T 30ad9ac7a90047a780832b99dca3b152 P 77e315cc324344468ac23d4fad7e40a9: Deleting consensus metadata
I20260619 06:40:13.017200 12391 catalog_manager.cc:4841] TS 74bcefa0f2744317b352f797378c0001 (127.7.146.193:42677): tablet 30ad9ac7a90047a780832b99dca3b152 (table test-table [id=707b180a31a24414892803aa12f647ae]) successfully deleted
I20260619 06:40:13.017709 12587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:13.018025 12392 catalog_manager.cc:4841] TS 77e315cc324344468ac23d4fad7e40a9 (127.7.146.194:32879): tablet 30ad9ac7a90047a780832b99dca3b152 (table test-table [id=707b180a31a24414892803aa12f647ae]) successfully deleted
I20260619 06:40:13.019558 12896 ts_tablet_manager.cc:1918] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:13.019696 12896 log.cc:1196] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/30ad9ac7a90047a780832b99dca3b152
I20260619 06:40:13.020067 12896 ts_tablet_manager.cc:1939] T 30ad9ac7a90047a780832b99dca3b152 P 5bf19f7723ab42b8bda3139abeb5dc6d: Deleting consensus metadata
I20260619 06:40:13.021765 12392 catalog_manager.cc:4841] TS 5bf19f7723ab42b8bda3139abeb5dc6d (127.7.146.195:40993): tablet 30ad9ac7a90047a780832b99dca3b152 (table test-table [id=707b180a31a24414892803aa12f647ae]) successfully deleted
I20260619 06:40:13.106003 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 12465
I20260619 06:40:13.120165 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 12596
I20260619 06:40:13.123145 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:13.121063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52528 (local address 127.7.146.194:32879)
0619 06:40:13.121265 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:13.121271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:13.121312 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:40:13.121401 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:13.121407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:13.121476 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:13.121585 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:13.122422 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:13.122428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:13.122433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:13.122453 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:13.122493 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:13.122508 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:13.122647 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:13.122753 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 06:40:13.122810 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:40:13.123922 12718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:13.231973 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 12596
I20260619 06:40:13.245092 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 12726
I20260619 06:40:13.247668 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:13.245842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.7.146.195:40993)
0619 06:40:13.246009 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:13.246013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:13.246031 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:40:13.246186 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:13.246191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:13.246270 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:13.246374 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:13.247163 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:13.247167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:13.247171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:13.247187 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:13.247218 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:13.247230 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:13.247337 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:13.247405 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 06:40:13.247450 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:40:13.248311 12848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:13.334132 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 12726
I20260619 06:40:13.346181 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 12375
I20260619 06:40:13.347285 12436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:13.450325 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 12375
2026-06-19T06:40:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2639 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 06:40:13.486498 7755 mini_oidc.cc:152] Starting JWKS server
I20260619 06:40:13.488998 7755 webserver.cc:460] Webserver started at https://127.0.0.1:46321/ using document root <none> and password file <none>
I20260619 06:40:13.489302 7755 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:40:13.490023 7755 webserver.cc:460] Webserver started at http://127.0.0.1:45717/ using document root <none> and password file <none>
2026-06-19T06:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:13Z Disabled control of system clock
I20260619 06:40:13.513036 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:45493
--webserver_interface=127.7.146.254
--webserver_port=0
--builtin_ntp_servers=127.7.146.212:34477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:45493
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46321/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/testchainca.cert with env {}
W20260619 06:40:13.821456 12908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:13.821736 12908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:13.821820 12908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:13.828984 12908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:13.829085 12908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:13.829130 12908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:13.829183 12908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:13.829224 12908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:40:13.829281 12908 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:40:13.829319 12908 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46321/jwks.json
I20260619 06:40:13.839151 12908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:45493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:45493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46321/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:13.840354 12908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:13.841616 12908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:13.852973 12913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:13.852880 12916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:13.852929 12914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:13.854310 12908 server_base.cc:1034] running on GCE node
I20260619 06:40:13.855091 12908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:13.856444 12908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:13.857628 12908 hybrid_clock.cc:648] HybridClock initialized: now 1781851213857585 us; error 37 us; skew 500 ppm
I20260619 06:40:13.857996 12908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:40:13.860110 12908 webserver.cc:460] Webserver started at http://127.7.146.254:35079/ using document root <none> and password file <none>
I20260619 06:40:13.860801 12908 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:13.860896 12908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:13.861137 12908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:13.863840 12908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "dba993f0696548eea6818292226401d0"
format_stamp: "Formatted at 2026-06-19 06:40:13 on dist-test-slave-qfk4"
server_key: "7e6b445ccff03ad4c4ce6bcc23c85612"
server_key_iv: "9dbc834a2eda1b0c6cab3e2b0bd8d914"
server_key_version: "encryptionkey@0"
I20260619 06:40:13.864550 12908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "dba993f0696548eea6818292226401d0"
format_stamp: "Formatted at 2026-06-19 06:40:13 on dist-test-slave-qfk4"
server_key: "7e6b445ccff03ad4c4ce6bcc23c85612"
server_key_iv: "9dbc834a2eda1b0c6cab3e2b0bd8d914"
server_key_version: "encryptionkey@0"
I20260619 06:40:13.869383 12908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:40:13.872622 12922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:13.873955 12908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:40:13.874085 12908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "dba993f0696548eea6818292226401d0"
format_stamp: "Formatted at 2026-06-19 06:40:13 on dist-test-slave-qfk4"
server_key: "7e6b445ccff03ad4c4ce6bcc23c85612"
server_key_iv: "9dbc834a2eda1b0c6cab3e2b0bd8d914"
server_key_version: "encryptionkey@0"
I20260619 06:40:13.874251 12908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:13.887190 12908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:13.887971 12908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:13.888223 12908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:13.905277 12908 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:45493
I20260619 06:40:13.905303 12973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:45493 every 8 connection(s)
I20260619 06:40:13.906607 12908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:13.909703 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 12908
I20260619 06:40:13.909885 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:13.910351 12974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:13.910518 7755 external_mini_cluster.cc:1428] Setting key 54416e76e5da10feeee441e609e27c38
I20260619 06:40:13.925063 12974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0: Bootstrap starting.
I20260619 06:40:13.925268 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:13.912331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.7.146.254:45493)
0619 06:40:13.913380 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:13.913399 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:13.913451 (+ 52us) server_negotiation.cc:407] Connection header received
0619 06:40:13.915173 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:13.915204 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:13.915879 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:13.916338 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:13.916925 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:13.918841 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:13.920136 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:13.921150 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:13.921342 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:13.922422 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:13.922432 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:13.922451 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:13.923169 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:13.923263 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:13.923296 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:13.923402 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:13.923612 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 06:40:13.923950 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":287,"threads_started":1}
I20260619 06:40:13.928309 12974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:13.928966 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:40:13.929391 12974 log.cc:824] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:13.932196 12974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0: No bootstrap required, opened a new log
I20260619 06:40:13.939477 12974 raft_consensus.cc:348] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba993f0696548eea6818292226401d0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45493 } }
I20260619 06:40:13.939821 12974 raft_consensus.cc:374] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:13.939894 12974 raft_consensus.cc:729] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba993f0696548eea6818292226401d0, State: Initialized, Role: FOLLOWER
I20260619 06:40:13.940387 12974 consensus_queue.cc:260] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba993f0696548eea6818292226401d0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45493 } }
I20260619 06:40:13.940621 12974 raft_consensus.cc:388] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:13.940723 12974 raft_consensus.cc:482] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:13.940837 12974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:13.942909 12974 raft_consensus.cc:504] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba993f0696548eea6818292226401d0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45493 } }
I20260619 06:40:13.943457 12974 leader_election.cc:302] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dba993f0696548eea6818292226401d0; no voters:
I20260619 06:40:13.944027 12974 leader_election.cc:288] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:13.944155 12979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:13.944813 12979 raft_consensus.cc:686] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [term 1 LEADER]: Becoming Leader. State: Replica: dba993f0696548eea6818292226401d0, State: Running, Role: LEADER
I20260619 06:40:13.945360 12979 consensus_queue.cc:237] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba993f0696548eea6818292226401d0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45493 } }
I20260619 06:40:13.945950 12974 sys_catalog.cc:564] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:13.948292 12981 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dba993f0696548eea6818292226401d0. Latest consensus state: current_term: 1 leader_uuid: "dba993f0696548eea6818292226401d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba993f0696548eea6818292226401d0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45493 } } }
I20260619 06:40:13.948621 12981 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:13.949350 12980 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dba993f0696548eea6818292226401d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba993f0696548eea6818292226401d0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45493 } } }
I20260619 06:40:13.949716 12980 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:13.950950 12986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:13.956269 12986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:13.965651 12986 catalog_manager.cc:1347] Generated new cluster ID: e6366827bec64201a7a491c33d4d6636
I20260619 06:40:13.965764 12986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:13.986511 12986 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:13.987802 12986 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:13.995509 12986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dba993f0696548eea6818292226401d0: Generated new TSK 0
I20260619 06:40:13.996610 12986 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:40:14.049613 13004 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 06:40:14.049791 13004 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:40:14.050592 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:14.022026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.7.146.254:45493)
0619 06:40:14.022230 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:14.022237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:14.022632 (+ 395us) server_negotiation.cc:407] Connection header received
0619 06:40:14.023009 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:14.023013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:14.023090 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:14.023209 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:40:14.023894 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:14.024679 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:14.026753 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:14.027196 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:14.027289 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:14.027925 (+ 636us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:40:14.049993 (+ 22068us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:40:14.050319 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 06:40:14.050714 12977 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 06:40:14.052389 13003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:14.021834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:45493 (local address 127.0.0.1:42900)
0619 06:40:14.022516 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:14.022537 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:14.022726 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:14.023256 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:14.023262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:14.023282 (+ 20us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:40:14.023636 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:14.023649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:14.024825 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:14.024831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:14.026592 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:14.026609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:14.027362 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:14.027369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:14.027566 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:14.027653 (+ 87us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:40:14.050274 (+ 22621us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:40:14.050634 (+ 360us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.146.254:45493: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":304,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.146.254:45493: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 06:40:14.054255 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 12908
I20260619 06:40:14.055603 12969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:14.148846 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 12908
2026-06-19T06:40:14Z chronyd exiting
I20260619 06:40:14.196087 7755 test_util.cc:175] -----------------------------------------------
I20260619 06:40:14.196218 7755 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851174422545-7755-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (714 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 06:40:14.199849 7755 mini_oidc.cc:152] Starting JWKS server
I20260619 06:40:14.201668 7755 webserver.cc:460] Webserver started at https://127.0.0.1:36669/ using document root <none> and password file <none>
I20260619 06:40:14.201861 7755 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:40:14.202455 7755 webserver.cc:460] Webserver started at http://127.0.0.1:33821/ using document root <none> and password file <none>
2026-06-19T06:40:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:14Z Disabled control of system clock
I20260619 06:40:14.226239 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:41759
--webserver_interface=127.7.146.254
--webserver_port=0
--builtin_ntp_servers=127.7.146.212:42523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:41759
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36669/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/testchainca.cert with env {}
W20260619 06:40:14.538857 13011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:14.539167 13011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:14.539299 13011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:14.546478 13011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:14.546597 13011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:14.546661 13011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:14.546741 13011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:14.546800 13011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:40:14.546891 13011 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:40:14.546942 13011 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36669/jwks.json
I20260619 06:40:14.556967 13011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:41759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:41759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36669/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:14.558245 13011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:14.559528 13011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:14.571020 13017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:14.571077 13016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:14.571050 13019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:14.571800 13011 server_base.cc:1034] running on GCE node
I20260619 06:40:14.572520 13011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:14.573933 13011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:14.575122 13011 hybrid_clock.cc:648] HybridClock initialized: now 1781851214575069 us; error 46 us; skew 500 ppm
I20260619 06:40:14.575459 13011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:40:14.577626 13011 webserver.cc:460] Webserver started at http://127.7.146.254:43997/ using document root <none> and password file <none>
I20260619 06:40:14.578266 13011 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:14.578372 13011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:14.578665 13011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:14.581444 13011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "08739d3aa0ae4b6e9f42346141db1eea"
format_stamp: "Formatted at 2026-06-19 06:40:14 on dist-test-slave-qfk4"
server_key: "fec9eb2997ed17fe2ad8ff7ace692a56"
server_key_iv: "c4921eb603f7acfbe99f9a3a2491c158"
server_key_version: "encryptionkey@0"
I20260619 06:40:14.582233 13011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "08739d3aa0ae4b6e9f42346141db1eea"
format_stamp: "Formatted at 2026-06-19 06:40:14 on dist-test-slave-qfk4"
server_key: "fec9eb2997ed17fe2ad8ff7ace692a56"
server_key_iv: "c4921eb603f7acfbe99f9a3a2491c158"
server_key_version: "encryptionkey@0"
I20260619 06:40:14.587148 13011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:40:14.590708 13025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:14.592180 13011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:40:14.592337 13011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "08739d3aa0ae4b6e9f42346141db1eea"
format_stamp: "Formatted at 2026-06-19 06:40:14 on dist-test-slave-qfk4"
server_key: "fec9eb2997ed17fe2ad8ff7ace692a56"
server_key_iv: "c4921eb603f7acfbe99f9a3a2491c158"
server_key_version: "encryptionkey@0"
I20260619 06:40:14.592537 13011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:14.640277 13011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:14.641191 13011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:14.641462 13011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:14.657994 13011 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:41759
I20260619 06:40:14.658018 13076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:41759 every 8 connection(s)
I20260619 06:40:14.659291 13011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:14.662915 13077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:14.663451 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 13011
I20260619 06:40:14.663618 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:14.664189 7755 external_mini_cluster.cc:1428] Setting key d4e3c103bdc73dd400f2d550e443007c
I20260619 06:40:14.676994 13077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea: Bootstrap starting.
I20260619 06:40:14.678920 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:14.666060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.7.146.254:41759)
0619 06:40:14.667119 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:14.667133 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:14.667182 (+ 49us) server_negotiation.cc:407] Connection header received
0619 06:40:14.669013 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:14.669044 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:14.669645 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:14.670090 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:14.670592 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:14.672373 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:14.673961 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:14.675015 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:14.675178 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:14.676178 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:14.676188 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:14.676206 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:14.676911 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:14.677031 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:14.677058 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:14.677159 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:14.677355 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 06:40:14.677667 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":314,"threads_started":1}
I20260619 06:40:14.680707 13077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:14.681901 13077 log.cc:824] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:14.682786 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:40:14.684782 13077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea: No bootstrap required, opened a new log
I20260619 06:40:14.692252 13077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08739d3aa0ae4b6e9f42346141db1eea" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 41759 } }
I20260619 06:40:14.692726 13077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:14.692806 13077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08739d3aa0ae4b6e9f42346141db1eea, State: Initialized, Role: FOLLOWER
I20260619 06:40:14.693343 13077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08739d3aa0ae4b6e9f42346141db1eea" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 41759 } }
I20260619 06:40:14.693558 13077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:14.693648 13077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:14.693764 13077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:14.695845 13077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08739d3aa0ae4b6e9f42346141db1eea" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 41759 } }
I20260619 06:40:14.696311 13077 leader_election.cc:302] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08739d3aa0ae4b6e9f42346141db1eea; no voters:
I20260619 06:40:14.696907 13077 leader_election.cc:288] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:14.697022 13082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:14.697752 13082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [term 1 LEADER]: Becoming Leader. State: Replica: 08739d3aa0ae4b6e9f42346141db1eea, State: Running, Role: LEADER
I20260619 06:40:14.698343 13082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08739d3aa0ae4b6e9f42346141db1eea" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 41759 } }
I20260619 06:40:14.699226 13077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:14.701092 13083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08739d3aa0ae4b6e9f42346141db1eea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08739d3aa0ae4b6e9f42346141db1eea" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 41759 } } }
I20260619 06:40:14.701486 13083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:14.704787 13084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08739d3aa0ae4b6e9f42346141db1eea. Latest consensus state: current_term: 1 leader_uuid: "08739d3aa0ae4b6e9f42346141db1eea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08739d3aa0ae4b6e9f42346141db1eea" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 41759 } } }
I20260619 06:40:14.705080 13084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:14.705204 13090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:14.710906 13090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:14.720186 13090 catalog_manager.cc:1347] Generated new cluster ID: 0250200c18f544d4bca4f5448cdf9e90
I20260619 06:40:14.720293 13090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:14.732936 13090 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:14.734174 13090 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:14.741650 13090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08739d3aa0ae4b6e9f42346141db1eea: Generated new TSK 0
I20260619 06:40:14.742444 13090 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:40:14.781759 13107 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 06:40:14.782064 13107 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:40:14.783237 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:14.762197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.7.146.254:41759)
0619 06:40:14.762402 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:14.762409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:14.762785 (+ 376us) server_negotiation.cc:407] Connection header received
0619 06:40:14.763087 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:14.763094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:14.763186 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:14.763361 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:40:14.763790 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:14.764910 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:14.766646 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:14.767302 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:14.767447 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:14.767948 (+ 501us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:40:14.782508 (+ 14560us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:40:14.782927 (+ 419us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54484: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 06:40:14.783363 13080 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54484: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 06:40:14.783370 13106 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:14.762066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:41759 (local address 127.0.0.1:54484)
0619 06:40:14.762686 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:14.762703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:14.762886 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:14.763372 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:14.763375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:14.763386 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:40:14.763632 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:14.763639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:14.765080 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:14.765087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:14.766487 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:14.766502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:14.767513 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:14.767518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:14.767642 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:14.767677 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:40:14.782769 (+ 15092us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:40:14.783039 (+ 270us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.146.254:41759: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":355,"threads_started":1}
I20260619 06:40:14.784752 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 13011
I20260619 06:40:14.785986 13072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:14.876984 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 13011
2026-06-19T06:40:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (727 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 06:40:14.927471 7755 mini_oidc.cc:152] Starting JWKS server
I20260619 06:40:14.929347 7755 webserver.cc:460] Webserver started at https://127.0.0.1:37731/ using document root <none> and password file <none>
I20260619 06:40:14.929589 7755 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:40:14.930208 7755 webserver.cc:460] Webserver started at http://127.0.0.1:45681/ using document root <none> and password file <none>
2026-06-19T06:40:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:14Z Disabled control of system clock
I20260619 06:40:14.954748 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:33363
--webserver_interface=127.7.146.254
--webserver_port=0
--builtin_ntp_servers=127.7.146.212:34825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:33363
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37731/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:40:15.266748 13114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:15.267057 13114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:15.267143 13114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:15.274405 13114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:15.274518 13114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:15.274554 13114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:15.274605 13114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:15.274646 13114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:40:15.274704 13114 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:40:15.274741 13114 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37731/jwks.json
I20260619 06:40:15.283957 13114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:33363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37731/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:15.285198 13114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:15.286868 13114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:15.297955 13119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:15.298027 13120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:15.297997 13122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:15.298841 13114 server_base.cc:1034] running on GCE node
I20260619 06:40:15.299660 13114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:15.301177 13114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:15.302405 13114 hybrid_clock.cc:648] HybridClock initialized: now 1781851215302353 us; error 43 us; skew 500 ppm
I20260619 06:40:15.302752 13114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:40:15.305075 13114 webserver.cc:460] Webserver started at http://127.7.146.254:43229/ using document root <none> and password file <none>
I20260619 06:40:15.305734 13114 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:15.305838 13114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:15.306145 13114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:15.309123 13114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "bd9ccda4c34a4b25a60570d4c173667c"
format_stamp: "Formatted at 2026-06-19 06:40:15 on dist-test-slave-qfk4"
server_key: "a1d769607092b1ac02fadd62a421a628"
server_key_iv: "7590208c0cf7f12ca8383b8d9007dd23"
server_key_version: "encryptionkey@0"
I20260619 06:40:15.309940 13114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "bd9ccda4c34a4b25a60570d4c173667c"
format_stamp: "Formatted at 2026-06-19 06:40:15 on dist-test-slave-qfk4"
server_key: "a1d769607092b1ac02fadd62a421a628"
server_key_iv: "7590208c0cf7f12ca8383b8d9007dd23"
server_key_version: "encryptionkey@0"
I20260619 06:40:15.315132 13114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:40:15.318629 13128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:15.320005 13114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:40:15.320180 13114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "bd9ccda4c34a4b25a60570d4c173667c"
format_stamp: "Formatted at 2026-06-19 06:40:15 on dist-test-slave-qfk4"
server_key: "a1d769607092b1ac02fadd62a421a628"
server_key_iv: "7590208c0cf7f12ca8383b8d9007dd23"
server_key_version: "encryptionkey@0"
I20260619 06:40:15.320393 13114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:15.352434 13114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:15.353291 13114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:15.353575 13114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:15.370175 13114 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:33363
I20260619 06:40:15.370199 13179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:33363 every 8 connection(s)
I20260619 06:40:15.371495 13114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:15.371870 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 13114
I20260619 06:40:15.372087 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:15.372869 7755 external_mini_cluster.cc:1428] Setting key 8bfd434a5ab89b8628d0f7488e0b8c02
I20260619 06:40:15.376040 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:15.387962 13183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:15.374725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.7.146.254:33363)
0619 06:40:15.375744 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:15.375758 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:15.375805 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:40:15.377933 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:15.377959 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:15.378548 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:15.378984 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:15.379514 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:15.381501 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:15.382762 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:15.383747 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:15.383941 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:15.385072 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:15.385083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:15.385101 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:15.385834 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:15.385932 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:15.385961 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:15.386098 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:15.386322 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 06:40:15.386669 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":311,"threads_started":1}
I20260619 06:40:15.389545 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c: Bootstrap starting.
I20260619 06:40:15.391995 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:40:15.392952 13180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:15.393925 13180 log.cc:824] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:15.396862 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c: No bootstrap required, opened a new log
I20260619 06:40:15.404258 13180 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9ccda4c34a4b25a60570d4c173667c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33363 } }
I20260619 06:40:15.404762 13180 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:15.404876 13180 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9ccda4c34a4b25a60570d4c173667c, State: Initialized, Role: FOLLOWER
I20260619 06:40:15.405390 13180 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9ccda4c34a4b25a60570d4c173667c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33363 } }
I20260619 06:40:15.405582 13180 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:15.405676 13180 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:15.405799 13180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:15.407811 13180 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9ccda4c34a4b25a60570d4c173667c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33363 } }
I20260619 06:40:15.408281 13180 leader_election.cc:302] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd9ccda4c34a4b25a60570d4c173667c; no voters:
I20260619 06:40:15.408797 13180 leader_election.cc:288] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:15.408910 13185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:15.409566 13185 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [term 1 LEADER]: Becoming Leader. State: Replica: bd9ccda4c34a4b25a60570d4c173667c, State: Running, Role: LEADER
I20260619 06:40:15.410072 13185 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9ccda4c34a4b25a60570d4c173667c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33363 } }
I20260619 06:40:15.411013 13180 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:15.412901 13187 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd9ccda4c34a4b25a60570d4c173667c. Latest consensus state: current_term: 1 leader_uuid: "bd9ccda4c34a4b25a60570d4c173667c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9ccda4c34a4b25a60570d4c173667c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33363 } } }
I20260619 06:40:15.412968 13186 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd9ccda4c34a4b25a60570d4c173667c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9ccda4c34a4b25a60570d4c173667c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33363 } } }
I20260619 06:40:15.413272 13187 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:15.413311 13186 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:15.417537 13190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:15.422814 13190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:15.433044 13190 catalog_manager.cc:1347] Generated new cluster ID: 36885e68af684c0f89be17c4fcdf3779
I20260619 06:40:15.433182 13190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:15.445300 13190 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:15.446631 13190 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:15.452522 13190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd9ccda4c34a4b25a60570d4c173667c: Generated new TSK 0
I20260619 06:40:15.453399 13190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:40:15.473496 13183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:15.465133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.7.146.254:33363)
0619 06:40:15.465377 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:15.465384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:15.465869 (+ 485us) server_negotiation.cc:407] Connection header received
0619 06:40:15.466087 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:15.466093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:15.466184 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:15.466353 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:40:15.466752 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:15.467859 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:15.469205 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:15.469899 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:15.470036 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:15.470522 (+ 486us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:40:15.472854 (+ 2332us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:40:15.473172 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58414: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":100}
W20260619 06:40:15.473623 13183 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58414: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 06:40:15.475181 13209 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:15.465133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33363 (local address 127.0.0.1:58414)
0619 06:40:15.465768 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:15.465784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:15.465914 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:15.466349 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:15.466352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:15.466361 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:40:15.466614 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:15.466622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:15.467989 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:15.467993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:15.469074 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:15.469086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:15.470090 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:15.470095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:15.470215 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:15.470249 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:40:15.473052 (+ 2803us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:40:15.474763 (+ 1711us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.146.254:33363: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":1437,"thread_start_us":346,"threads_started":1}
I20260619 06:40:15.476867 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 13114
I20260619 06:40:15.478065 13175 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 06:40:15.485072 13210 webserver.cc:533] Webserver: sslize failed:
E20260619 06:40:15.485167 13210 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:40:15.567812 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 13114
2026-06-19T06:40:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (691 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T06:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:15Z Disabled control of system clock
I20260619 06:40:15.641757 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:33847
--webserver_interface=127.7.146.254
--webserver_port=0
--builtin_ntp_servers=127.7.146.212:36107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:33847
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/testchainca.cert with env {}
W20260619 06:40:15.951251 13215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:15.951570 13215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:15.951654 13215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:15.958896 13215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:15.959018 13215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:15.959056 13215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:15.959111 13215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:15.959154 13215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:40:15.959211 13215 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:40:15.959249 13215 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 06:40:15.969118 13215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:36107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:33847
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:33847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:15.970414 13215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:15.971648 13215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:15.982901 13223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:15.982939 13220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:15.982960 13221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:15.983666 13215 server_base.cc:1034] running on GCE node
I20260619 06:40:15.984393 13215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:15.985826 13215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:15.987012 13215 hybrid_clock.cc:648] HybridClock initialized: now 1781851215986965 us; error 40 us; skew 500 ppm
I20260619 06:40:15.987330 13215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:40:15.989451 13215 webserver.cc:460] Webserver started at http://127.7.146.254:35097/ using document root <none> and password file <none>
I20260619 06:40:15.990103 13215 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:15.990196 13215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:15.990481 13215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:15.993341 13215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "0caff8877498486b8e0500d5984a23a7"
format_stamp: "Formatted at 2026-06-19 06:40:15 on dist-test-slave-qfk4"
server_key: "03fc38352829f04dac835015f909bb2a"
server_key_iv: "6593467776a7d625953c95db85ea6360"
server_key_version: "encryptionkey@0"
I20260619 06:40:15.994105 13215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "0caff8877498486b8e0500d5984a23a7"
format_stamp: "Formatted at 2026-06-19 06:40:15 on dist-test-slave-qfk4"
server_key: "03fc38352829f04dac835015f909bb2a"
server_key_iv: "6593467776a7d625953c95db85ea6360"
server_key_version: "encryptionkey@0"
I20260619 06:40:15.998917 13215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:40:16.002224 13229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:16.003539 13215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:40:16.003698 13215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "0caff8877498486b8e0500d5984a23a7"
format_stamp: "Formatted at 2026-06-19 06:40:15 on dist-test-slave-qfk4"
server_key: "03fc38352829f04dac835015f909bb2a"
server_key_iv: "6593467776a7d625953c95db85ea6360"
server_key_version: "encryptionkey@0"
I20260619 06:40:16.003891 13215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:16.016213 13215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:16.017032 13215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:16.017287 13215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:16.033584 13215 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:33847
I20260619 06:40:16.033604 13280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:33847 every 8 connection(s)
I20260619 06:40:16.034871 13215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:16.038048 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 13215
I20260619 06:40:16.038213 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:16.038570 13281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:16.038872 7755 external_mini_cluster.cc:1428] Setting key 29d6121f0203da6786a97a3fd3239100
I20260619 06:40:16.050591 13281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7: Bootstrap starting.
I20260619 06:40:16.051357 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:16.040693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.7.146.254:33847)
0619 06:40:16.041619 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:16.041630 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:16.041671 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:40:16.043186 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:16.043206 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:16.043675 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:16.044056 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:16.044638 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:16.045881 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:16.047129 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:16.047807 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:16.047975 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:16.049083 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:16.049090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:16.049104 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:16.049619 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:16.049690 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:16.049710 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:16.049811 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:16.050029 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 06:40:16.050313 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":295,"threads_started":1}
I20260619 06:40:16.053944 13281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:16.054965 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:40:16.055047 13281 log.cc:824] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:16.057744 13281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7: No bootstrap required, opened a new log
I20260619 06:40:16.064549 13281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caff8877498486b8e0500d5984a23a7" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33847 } }
I20260619 06:40:16.064944 13281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:16.065042 13281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0caff8877498486b8e0500d5984a23a7, State: Initialized, Role: FOLLOWER
I20260619 06:40:16.065605 13281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caff8877498486b8e0500d5984a23a7" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33847 } }
I20260619 06:40:16.065796 13281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:16.065886 13281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:16.066000 13281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:16.068001 13281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caff8877498486b8e0500d5984a23a7" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33847 } }
I20260619 06:40:16.068471 13281 leader_election.cc:302] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0caff8877498486b8e0500d5984a23a7; no voters:
I20260619 06:40:16.069190 13281 leader_election.cc:288] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:16.069308 13286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:16.069990 13286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [term 1 LEADER]: Becoming Leader. State: Replica: 0caff8877498486b8e0500d5984a23a7, State: Running, Role: LEADER
I20260619 06:40:16.070595 13286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caff8877498486b8e0500d5984a23a7" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33847 } }
I20260619 06:40:16.071563 13281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:16.073520 13287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0caff8877498486b8e0500d5984a23a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caff8877498486b8e0500d5984a23a7" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33847 } } }
I20260619 06:40:16.073438 13288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0caff8877498486b8e0500d5984a23a7. Latest consensus state: current_term: 1 leader_uuid: "0caff8877498486b8e0500d5984a23a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caff8877498486b8e0500d5984a23a7" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33847 } } }
I20260619 06:40:16.073952 13288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:16.073967 13287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:16.075832 13293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:16.081554 13293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:16.090010 13293 catalog_manager.cc:1347] Generated new cluster ID: 6075632ed4454871ac23e405f73ff86f
I20260619 06:40:16.090126 13293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:16.111828 13293 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:16.113193 13293 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:16.120888 13293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0caff8877498486b8e0500d5984a23a7: Generated new TSK 0
I20260619 06:40:16.122010 13293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:40:16.617125 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:16.144982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.7.146.254:33847)
0619 06:40:16.145192 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:16.145199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:16.145580 (+ 381us) server_negotiation.cc:407] Connection header received
0619 06:40:16.145814 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:16.145819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:16.145896 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:16.146028 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:40:16.146567 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:16.147273 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:16.149297 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:16.149743 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:16.149831 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:16.150365 (+ 534us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:40:16.616480 (+466115us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:40:16.616884 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36978: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 06:40:16.617249 13284 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36978: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 06:40:16.617401 13310 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:16.144855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33847 (local address 127.0.0.1:36978)
0619 06:40:16.145467 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:16.145491 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:16.145644 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:16.146061 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:16.146067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:16.146079 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:40:16.146412 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:16.146424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:16.147412 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:16.147418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:16.149160 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:16.149177 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:16.149899 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:16.149905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:16.150100 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:16.150146 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:40:16.616809 (+466663us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:40:16.617053 (+ 244us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.146.254:33847: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":323,"threads_started":1}
I20260619 06:40:16.618480 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 13215
I20260619 06:40:16.619671 13276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:16.721132 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 13215
2026-06-19T06:40:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1134 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (391 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (397 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:40:17 dist-test-slave-qfk4 krb5kdc[13325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:40:17 dist-test-slave-qfk4 krb5kdc[13325](info): set up 2 sockets
Jun 19 06:40:17 dist-test-slave-qfk4 krb5kdc[13325](info): commencing operation
krb5kdc: starting...
W20260619 06:40:18.517061 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:40:18 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:40:18.708506 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:35753
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:35753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:19.019789 13338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:19.020084 13338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:19.020195 13338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:19.027514 13338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:19.027622 13338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:19.027675 13338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:19.027755 13338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:19.027808 13338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:19.037835 13338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:35753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:35753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:19.039167 13338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:19.040459 13338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:19.052096 13344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:19.052096 13346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:19.052415 13343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:19.052811 13338 server_base.cc:1034] running on GCE node
I20260619 06:40:19.053556 13338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:19.054920 13338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:19.056138 13338 hybrid_clock.cc:648] HybridClock initialized: now 1781851219056084 us; error 41 us; skew 500 ppm
I20260619 06:40:19.056466 13338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:19 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851219, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:19.059441 13338 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:40:19.060973 13338 webserver.cc:460] Webserver started at http://127.7.146.254:41161/ using document root <none> and password file <none>
I20260619 06:40:19.061635 13338 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:19.061738 13338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:19.062042 13338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:19.065052 13338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "6642e9891ad743948ad3deb5ef82eb19"
format_stamp: "Formatted at 2026-06-19 06:40:19 on dist-test-slave-qfk4"
server_key: "65a7429947937b86b439615ad4879495"
server_key_iv: "78befef433ea5da9573a70a82f658c8e"
server_key_version: "encryptionkey@0"
I20260619 06:40:19.065814 13338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "6642e9891ad743948ad3deb5ef82eb19"
format_stamp: "Formatted at 2026-06-19 06:40:19 on dist-test-slave-qfk4"
server_key: "65a7429947937b86b439615ad4879495"
server_key_iv: "78befef433ea5da9573a70a82f658c8e"
server_key_version: "encryptionkey@0"
I20260619 06:40:19.070794 13338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:40:19.074252 13353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:19.075587 13338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:40:19.075732 13338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "6642e9891ad743948ad3deb5ef82eb19"
format_stamp: "Formatted at 2026-06-19 06:40:19 on dist-test-slave-qfk4"
server_key: "65a7429947937b86b439615ad4879495"
server_key_iv: "78befef433ea5da9573a70a82f658c8e"
server_key_version: "encryptionkey@0"
I20260619 06:40:19.075919 13338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:19.101670 13338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:19.106262 13338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:19.106539 13338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:19.123545 13338 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:35753
I20260619 06:40:19.123569 13404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:35753 every 8 connection(s)
I20260619 06:40:19.124908 13338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:19.126988 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 13338
I20260619 06:40:19.127188 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:19.127962 7755 external_mini_cluster.cc:1428] Setting key 4f8d68b36db951ac9e134b70feadbebf
I20260619 06:40:19.129334 13405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:19.142805 13405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19: Bootstrap starting.
Jun 19 06:40:19 dist-test-slave-qfk4 krb5kdc[13325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:19.146059 13405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:19.147087 13405 log.cc:824] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:19.149734 13405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19: No bootstrap required, opened a new log
I20260619 06:40:19.151038 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:19.130023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.7.146.254:35753)
0619 06:40:19.130877 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:19.130889 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:19.130932 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:40:19.132647 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:19.132671 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:19.133124 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:19.133479 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:19.135257 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:19.136821 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:19.138505 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:19.139298 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:19.139439 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:19.143922 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:19.143931 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:19.143946 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:19.143993 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:19.147122 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:19.147837 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:19.147844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:19.147850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:19.148034 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:19.148403 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:19.148406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:19.148408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:19.148951 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:19.149200 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:19.149633 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 06:40:19.150082 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":418,"threads_started":1}
I20260619 06:40:19.156845 13405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642e9891ad743948ad3deb5ef82eb19" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35753 } }
I20260619 06:40:19.157176 13405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:19.157260 13405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6642e9891ad743948ad3deb5ef82eb19, State: Initialized, Role: FOLLOWER
I20260619 06:40:19.157812 13405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642e9891ad743948ad3deb5ef82eb19" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35753 } }
I20260619 06:40:19.158020 13405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:19.158133 13405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:19.158286 13405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:19.160272 13405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642e9891ad743948ad3deb5ef82eb19" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35753 } }
I20260619 06:40:19.160787 13405 leader_election.cc:302] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6642e9891ad743948ad3deb5ef82eb19; no voters:
I20260619 06:40:19.161387 13405 leader_election.cc:288] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:19.161505 13410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:19.162130 13410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [term 1 LEADER]: Becoming Leader. State: Replica: 6642e9891ad743948ad3deb5ef82eb19, State: Running, Role: LEADER
I20260619 06:40:19.162760 13410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642e9891ad743948ad3deb5ef82eb19" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35753 } }
I20260619 06:40:19.163508 13405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:19.166332 13412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6642e9891ad743948ad3deb5ef82eb19. Latest consensus state: current_term: 1 leader_uuid: "6642e9891ad743948ad3deb5ef82eb19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642e9891ad743948ad3deb5ef82eb19" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35753 } } }
I20260619 06:40:19.167443 13412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:19.165755 13411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6642e9891ad743948ad3deb5ef82eb19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642e9891ad743948ad3deb5ef82eb19" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35753 } } }
I20260619 06:40:19.169031 13411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19 [sys.catalog]: This master's current role is: LEADER
W20260619 06:40:19.174654 13426 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:40:19.174762 13426 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:40:19.174890 13421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:19.176610 13421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:19.185616 13421 catalog_manager.cc:1347] Generated new cluster ID: 08ae17d0ea4d4d319f231f476768217d
I20260619 06:40:19.185729 13421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:19.196848 13421 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:19.198122 13421 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:19.207175 13421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6642e9891ad743948ad3deb5ef82eb19: Generated new TSK 0
I20260619 06:40:19.208092 13421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:40:19.267695 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35753
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:19.587661 13431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:19.587946 13431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:19.588064 13431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:19.595382 13431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:19.595496 13431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:19.595659 13431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:40:19.605381 13431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:19.606604 13431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:19.607932 13431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:19.620447 13436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:19.620533 13439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:19.620426 13437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:19.621163 13431 server_base.cc:1034] running on GCE node
I20260619 06:40:19.621708 13431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:19.622681 13431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:19.623909 13431 hybrid_clock.cc:648] HybridClock initialized: now 1781851219623852 us; error 54 us; skew 500 ppm
I20260619 06:40:19.624269 13431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:19 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851219, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:19.627310 13431 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:40:19.628747 13431 webserver.cc:460] Webserver started at http://127.7.146.193:40001/ using document root <none> and password file <none>
I20260619 06:40:19.629369 13431 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:19.629464 13431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:19.629705 13431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:19.632475 13431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "f11f7abb8cd540a38da93552f6416f92"
format_stamp: "Formatted at 2026-06-19 06:40:19 on dist-test-slave-qfk4"
server_key: "17857037181d08e02541a07cdae7130c"
server_key_iv: "fa744205e6d4d7dba437f918ce1298f5"
server_key_version: "encryptionkey@0"
I20260619 06:40:19.633257 13431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "f11f7abb8cd540a38da93552f6416f92"
format_stamp: "Formatted at 2026-06-19 06:40:19 on dist-test-slave-qfk4"
server_key: "17857037181d08e02541a07cdae7130c"
server_key_iv: "fa744205e6d4d7dba437f918ce1298f5"
server_key_version: "encryptionkey@0"
I20260619 06:40:19.638139 13431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:40:19.641511 13446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:19.642952 13431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:40:19.643108 13431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "f11f7abb8cd540a38da93552f6416f92"
format_stamp: "Formatted at 2026-06-19 06:40:19 on dist-test-slave-qfk4"
server_key: "17857037181d08e02541a07cdae7130c"
server_key_iv: "fa744205e6d4d7dba437f918ce1298f5"
server_key_version: "encryptionkey@0"
I20260619 06:40:19.643293 13431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:19.666549 13431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:19.670933 13431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:19.671197 13431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:19.672156 13431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:19.673539 13431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:19.673619 13431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:19.673736 13431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:19.673787 13431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:19.704084 13431 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:40705
I20260619 06:40:19.704141 13558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:40705 every 8 connection(s)
I20260619 06:40:19.705379 13431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:40:19.709900 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 13431
I20260619 06:40:19.710095 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:40:19.710686 7755 external_mini_cluster.cc:1428] Setting key 3daf5a1d323722ca0f6b8a56f0cd3926
Jun 19 06:40:19 dist-test-slave-qfk4 krb5kdc[13325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851219, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:19.726976 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:19.708193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:56031 (local address 127.7.146.254:35753)
0619 06:40:19.708661 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:19.708666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:19.709677 (+ 1011us) server_negotiation.cc:407] Connection header received
0619 06:40:19.711573 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:19.711580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:19.711668 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:19.711774 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:19.713367 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:19.714406 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:19.716117 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:19.716679 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:19.716798 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:19.720818 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:19.720823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:19.720826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:19.720948 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:19.723404 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:19.724403 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:19.724408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:19.724411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:19.724551 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:19.725096 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:19.725100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:19.725102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:19.725449 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:19.725566 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:19.726526 (+ 960us) server_negotiation.cc:299] Negotiation successful
0619 06:40:19.726708 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":227,"threads_started":1}
I20260619 06:40:19.728422 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:19.708706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:35753 (local address 127.7.146.193:56031)
0619 06:40:19.709527 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:19.709571 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:19.711290 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:19.712020 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:19.712029 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:19.712637 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:19.713207 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:19.713225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:19.714579 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:19.714583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:19.715967 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:19.715981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:19.716934 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:19.716940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:19.717152 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:19.717801 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:19.717839 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:19.720548 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:19.723565 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:19.723575 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:19.723596 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:19.724236 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:19.724706 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:19.724710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:19.724713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:19.724969 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:19.725584 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:19.725594 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:19.726072 (+ 478us) client_negotiation.cc:769] Sending connection context
0619 06:40:19.726396 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 06:40:19.726983 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":293,"threads_started":1}
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
I20260619 06:40:19.729956 13559 heartbeater.cc:347] Connected to a master server at 127.7.146.254:35753
I20260619 06:40:19.730249 13559 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:19.730981 13559 heartbeater.cc:511] Master 127.7.146.254:35753 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
I20260619 06:40:19.733070 13370 ts_manager.cc:194] Registered new tserver with Master: f11f7abb8cd540a38da93552f6416f92 (127.7.146.193:40705)
I20260619 06:40:19.734475 13370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:56031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:40:19.753743 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35753
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:20.060508 13565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:20.060873 13565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:20.060981 13565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:20.068535 13565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:20.068691 13565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:20.068813 13565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:40:20.078577 13565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:20.079811 13565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:20.081287 13565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:20.093712 13570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:20.093813 13571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:20.093935 13573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:20.094732 13565 server_base.cc:1034] running on GCE node
I20260619 06:40:20.095398 13565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:20.096354 13565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:20.097572 13565 hybrid_clock.cc:648] HybridClock initialized: now 1781851220097519 us; error 41 us; skew 500 ppm
I20260619 06:40:20.097894 13565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:20.101154 13565 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:40:20.102591 13565 webserver.cc:460] Webserver started at http://127.7.146.194:34415/ using document root <none> and password file <none>
I20260619 06:40:20.103255 13565 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:20.103367 13565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:20.103659 13565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:20.106534 13565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "d06641264f784106be723848e8b6ccd2"
format_stamp: "Formatted at 2026-06-19 06:40:20 on dist-test-slave-qfk4"
server_key: "bb33beaf08107efd95b383134c10cc89"
server_key_iv: "25a059eeaddd442c8ed7d0ddd0a51da9"
server_key_version: "encryptionkey@0"
I20260619 06:40:20.107280 13565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "d06641264f784106be723848e8b6ccd2"
format_stamp: "Formatted at 2026-06-19 06:40:20 on dist-test-slave-qfk4"
server_key: "bb33beaf08107efd95b383134c10cc89"
server_key_iv: "25a059eeaddd442c8ed7d0ddd0a51da9"
server_key_version: "encryptionkey@0"
I20260619 06:40:20.112558 13565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:40:20.116392 13580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:20.118043 13565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:20.118211 13565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "d06641264f784106be723848e8b6ccd2"
format_stamp: "Formatted at 2026-06-19 06:40:20 on dist-test-slave-qfk4"
server_key: "bb33beaf08107efd95b383134c10cc89"
server_key_iv: "25a059eeaddd442c8ed7d0ddd0a51da9"
server_key_version: "encryptionkey@0"
I20260619 06:40:20.118410 13565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:20.139482 13565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:20.143946 13565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:20.144219 13565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:20.145265 13565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:20.146544 13565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:20.146622 13565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:20.146736 13565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:20.146785 13565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:20.178143 13565 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:40053
I20260619 06:40:20.178185 13692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:40053 every 8 connection(s)
I20260619 06:40:20.179347 13565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:40:20.181540 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 13565
I20260619 06:40:20.181690 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:40:20.182178 7755 external_mini_cluster.cc:1428] Setting key 91199485223a54d7bf99a939663ae6a3
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:20.200824 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:20.181847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:58991 (local address 127.7.146.254:35753)
0619 06:40:20.182051 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:20.182058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:20.183397 (+ 1339us) server_negotiation.cc:407] Connection header received
0619 06:40:20.185208 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:20.185214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:20.185296 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:20.185395 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:20.186923 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.187903 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.189793 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.190219 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.190317 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:20.194368 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:20.194374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:20.194377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:20.194415 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:20.197176 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.198200 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.198205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.198208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.198346 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.198882 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.198886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.198889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.199230 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:20.199367 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:20.200263 (+ 896us) server_negotiation.cc:299] Negotiation successful
0619 06:40:20.200461 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
I20260619 06:40:20.201997 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:20.182372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:35753 (local address 127.7.146.194:58991)
0619 06:40:20.183228 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:20.183283 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:20.184935 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:20.185600 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:20.185609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:20.186181 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:20.186756 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:20.186779 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.188065 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.188069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:20.189634 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:20.189650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.190434 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.190440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:20.190717 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:20.191429 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:20.191457 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:20.194088 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:20.197339 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:20.197348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:20.197368 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:20.198012 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:20.198460 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:20.198465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:20.198469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:20.198725 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:20.199373 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:20.199382 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:20.199852 (+ 470us) client_negotiation.cc:769] Sending connection context
0619 06:40:20.200181 (+ 329us) client_negotiation.cc:240] Negotiation successful
0619 06:40:20.200813 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":309,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
I20260619 06:40:20.203397 13693 heartbeater.cc:347] Connected to a master server at 127.7.146.254:35753
I20260619 06:40:20.203676 13693 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:20.204380 13693 heartbeater.cc:511] Master 127.7.146.254:35753 requested a full tablet report, sending...
I20260619 06:40:20.205833 13370 ts_manager.cc:194] Registered new tserver with Master: d06641264f784106be723848e8b6ccd2 (127.7.146.194:40053)
I20260619 06:40:20.206634 13370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:58991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:40:20.227385 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35753
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:20.535485 13698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:20.535785 13698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:20.535912 13698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:20.543334 13698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:20.543448 13698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:20.543593 13698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:40:20.553548 13698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:20.554811 13698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:20.556099 13698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:20.569242 13704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:20.569211 13703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:20.569307 13706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:20.570102 13698 server_base.cc:1034] running on GCE node
I20260619 06:40:20.570708 13698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:20.571609 13698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:20.572795 13698 hybrid_clock.cc:648] HybridClock initialized: now 1781851220572731 us; error 58 us; skew 500 ppm
I20260619 06:40:20.573100 13698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:20.576051 13698 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:40:20.577504 13698 webserver.cc:460] Webserver started at http://127.7.146.195:35733/ using document root <none> and password file <none>
I20260619 06:40:20.578161 13698 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:20.578269 13698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:20.578547 13698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:20.581444 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "3fcf5bd5832c43d18dfbbcbb6ba75772"
format_stamp: "Formatted at 2026-06-19 06:40:20 on dist-test-slave-qfk4"
server_key: "eae34e212bae32c3ea7bb9abd6b832f0"
server_key_iv: "031b5fae9ee774e0e67dbda9df50890e"
server_key_version: "encryptionkey@0"
I20260619 06:40:20.582167 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "3fcf5bd5832c43d18dfbbcbb6ba75772"
format_stamp: "Formatted at 2026-06-19 06:40:20 on dist-test-slave-qfk4"
server_key: "eae34e212bae32c3ea7bb9abd6b832f0"
server_key_iv: "031b5fae9ee774e0e67dbda9df50890e"
server_key_version: "encryptionkey@0"
I20260619 06:40:20.586964 13698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:40:20.590370 13713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:20.591698 13698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:40:20.591861 13698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "3fcf5bd5832c43d18dfbbcbb6ba75772"
format_stamp: "Formatted at 2026-06-19 06:40:20 on dist-test-slave-qfk4"
server_key: "eae34e212bae32c3ea7bb9abd6b832f0"
server_key_iv: "031b5fae9ee774e0e67dbda9df50890e"
server_key_version: "encryptionkey@0"
I20260619 06:40:20.592064 13698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:20.615818 13698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:20.620414 13698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:20.620736 13698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:20.621822 13698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:20.623287 13698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:20.623379 13698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:20.623507 13698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:20.623565 13698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:20.653589 13698 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:40967
I20260619 06:40:20.653636 13825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:40967 every 8 connection(s)
I20260619 06:40:20.654778 13698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:40:20.655412 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 13698
I20260619 06:40:20.655602 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:40:20.656126 7755 external_mini_cluster.cc:1428] Setting key c0c9640b018418e9c0519381fc9218da
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:20.674053 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:20.657323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:56877 (local address 127.7.146.254:35753)
0619 06:40:20.657475 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:20.657479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:20.658790 (+ 1311us) server_negotiation.cc:407] Connection header received
0619 06:40:20.660748 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:20.660754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:20.660833 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:20.660912 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:20.662387 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.663109 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.664689 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.665113 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.665223 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:20.668700 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:20.668706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:20.668709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:20.668743 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:20.671203 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.671891 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.671895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.671897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.672012 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.672400 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.672405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.672408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.672718 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:20.672855 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:20.673512 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 06:40:20.673722 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:40:20.675019 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:20.657812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:35753 (local address 127.7.146.195:56877)
0619 06:40:20.658612 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:20.658667 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:20.660427 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:20.661154 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:20.661163 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:20.661654 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:20.662221 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:20.662239 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.663245 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.663249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:20.664546 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:20.664557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.665317 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.665323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:20.665552 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:20.666192 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:20.666214 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:20.668483 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:20.671327 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:20.671335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:20.671352 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:20.671775 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:20.672108 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:20.672113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:20.672117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:20.672291 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:20.672847 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:20.672855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:20.673172 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 06:40:20.673405 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 06:40:20.673937 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":523,"threads_started":1}
I20260619 06:40:20.676329 13826 heartbeater.cc:347] Connected to a master server at 127.7.146.254:35753
I20260619 06:40:20.676635 13826 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:20.677376 13826 heartbeater.cc:511] Master 127.7.146.254:35753 requested a full tablet report, sending...
I20260619 06:40:20.678819 13370 ts_manager.cc:194] Registered new tserver with Master: 3fcf5bd5832c43d18dfbbcbb6ba75772 (127.7.146.195:40967)
I20260619 06:40:20.679616 13370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:56877
I20260619 06:40:20.683924 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/test-admin.keytab.
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:20.708441 7755 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 06:40:20.713162 13833 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:20.710885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.7.146.193:40705)
0619 06:40:20.711424 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:20.711431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:20.711457 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:40:20.711599 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:20.711606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:20.711806 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:20.711969 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:20.712760 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58666: BlockingRecv error: recv got EOF from 127.0.0.1:58666 (error 108)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":286,"threads_started":1}
W20260619 06:40:20.713163 7755 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:20.714624 7755 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:20.715874 7755 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:40:20.730741 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:20.717448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.7.146.193:40705)
0619 06:40:20.717631 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:20.717637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:20.717820 (+ 183us) server_negotiation.cc:407] Connection header received
0619 06:40:20.718031 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:20.718035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:20.718093 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:20.718172 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:20.719123 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.720076 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.721318 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.721868 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.722045 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:20.725381 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:20.725391 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:20.725406 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:20.725444 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:20.728061 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.728745 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.728754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.728757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.728886 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.729224 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.729227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.729229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.729580 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:20.729835 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:20.730220 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 06:40:20.730468 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 06:40:20.736425 13833 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:20.734812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.7.146.193:40705)
0619 06:40:20.735029 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:20.735035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:20.735298 (+ 263us) server_negotiation.cc:407] Connection header received
0619 06:40:20.735414 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:20.735418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:20.735487 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:20.735565 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:20.736207 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58682: BlockingRecv error: recv got EOF from 127.0.0.1:58682 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 06:40:20.736837 7755 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5cc)
I20260619 06:40:20.737609 13559 heartbeater.cc:503] Master 127.7.146.254:35753 was elected leader, sending a full tablet report...
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:20.738500 7755 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:20.739820 7755 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:40:20.753162 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:20.741134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.7.146.193:40705)
0619 06:40:20.741310 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:20.741316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:20.741485 (+ 169us) server_negotiation.cc:407] Connection header received
0619 06:40:20.741612 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:20.741615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:20.741670 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:20.741743 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:20.742624 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.743362 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.744476 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.744901 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.745012 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:20.748455 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:20.748460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:20.748464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:20.748493 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:20.750832 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.751415 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.751419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.751427 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.751544 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.751825 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.751828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.751830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.752110 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:20.752298 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:20.752702 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 06:40:20.752882 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:40:20.757545 13833 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:20.755810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.7.146.193:40705)
0619 06:40:20.756009 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:20.756015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:20.756325 (+ 310us) server_negotiation.cc:407] Connection header received
0619 06:40:20.756469 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:20.756473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:20.756542 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:20.756654 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:20.757327 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58704: BlockingRecv error: recv got EOF from 127.0.0.1:58704 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 06:40:20.757720 7755 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851174422545-7755-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:20.759140 7755 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:20.760375 7755 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:40:20.774300 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:20.761767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.7.146.193:40705)
0619 06:40:20.761946 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:20.761951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:20.762122 (+ 171us) server_negotiation.cc:407] Connection header received
0619 06:40:20.762252 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:20.762256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:20.762312 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:20.762396 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:20.763361 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.764207 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.765533 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.765952 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.766042 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:20.769496 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:20.769501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:20.769504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:20.769535 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:20.771909 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.772508 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.772513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.772516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.772712 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.773041 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.773045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.773047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.773365 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:20.773609 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:20.773795 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 06:40:20.774017 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:40:20.776024 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 13431
I20260619 06:40:20.788002 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:20.776748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.7.146.193:40705)
0619 06:40:20.776905 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:20.776911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:20.777200 (+ 289us) server_negotiation.cc:407] Connection header received
0619 06:40:20.777354 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:20.777359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:20.777424 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:20.777521 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:20.778478 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.779164 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.780653 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.781091 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.781199 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:20.783321 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:20.783326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:20.783329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:20.783360 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:20.785842 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.786460 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.786464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.786467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.786589 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.786862 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.786864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.786866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.787133 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:20.787329 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:20.787554 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 06:40:20.787709 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:40:20.788522 13554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:20.870265 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 13431
I20260619 06:40:20.883759 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 13565
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:40:20.899686 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:20.884754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.7.146.194:40053)
0619 06:40:20.885312 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:20.885322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:20.885350 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:40:20.885410 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:20.885417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:20.885641 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:20.885795 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:20.886891 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.887934 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.889240 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.889842 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.889971 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:20.893835 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:20.893846 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:20.893864 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:20.893915 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:20.896759 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.897598 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.897605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.897607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.897728 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:20.898149 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:20.898154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:20.898156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:20.898555 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:20.898803 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:20.899168 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 06:40:20.899390 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":286,"threads_started":1}
I20260619 06:40:20.900952 13688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:20.980532 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 13565
I20260619 06:40:20.992647 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 13698
Jun 19 06:40:20 dist-test-slave-qfk4 krb5kdc[13325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:40:21.009064 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:20.993623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.7.146.195:40967)
0619 06:40:20.994239 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:20.994246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:20.994277 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:40:20.994348 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:20.994355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:20.994564 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:20.994707 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:20.995817 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.997088 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.998357 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:20.998959 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:20.999069 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:21.003094 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:21.003109 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:21.003123 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:21.003163 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:21.006046 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:21.006829 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:21.006837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:21.006839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:21.006976 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:21.007425 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:21.007429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:21.007431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:21.007819 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:21.008104 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:21.008386 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 06:40:21.008723 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":294,"threads_started":1}
I20260619 06:40:21.010303 13821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:21.091698 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 13698
I20260619 06:40:21.103560 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 13338
I20260619 06:40:21.104821 13400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:21.207691 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 13338
2026-06-19T06:40:21Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3700 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:40:21 dist-test-slave-qfk4 krb5kdc[13853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:40:21 dist-test-slave-qfk4 krb5kdc[13853](info): set up 2 sockets
Jun 19 06:40:21 dist-test-slave-qfk4 krb5kdc[13853](info): commencing operation
krb5kdc: starting...
W20260619 06:40:22.213083 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:40:22 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.254.keytab.
Entry for principal oryx/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.254.keytab.
I20260619 06:40:22.404812 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:33983
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.254.keytab
--principal=oryx/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:33983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:22.711658 13866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:22.711952 13866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:22.712066 13866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:22.719209 13866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:22.719321 13866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:22.719381 13866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:22.719460 13866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:22.719516 13866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:22.729238 13866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:33983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.254.keytab
--principal=oryx/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:33983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:22.730494 13866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:22.731717 13866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:22.742367 13874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:22.742491 13871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:22.742416 13872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:22.743444 13866 server_base.cc:1034] running on GCE node
I20260619 06:40:22.744175 13866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:22.745560 13866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:22.746766 13866 hybrid_clock.cc:648] HybridClock initialized: now 1781851222746721 us; error 41 us; skew 500 ppm
I20260619 06:40:22.747093 13866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:22 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851222, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:22.752983 13866 init.cc:377] Logged in from keytab as oryx/127.7.146.254@KRBTEST.COM (short username oryx)
I20260619 06:40:22.754346 13866 webserver.cc:460] Webserver started at http://127.7.146.254:46809/ using document root <none> and password file <none>
I20260619 06:40:22.755036 13866 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:22.755136 13866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:22.755383 13866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:22.758291 13866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "871d0d204466410e8683d77affe62f3a"
format_stamp: "Formatted at 2026-06-19 06:40:22 on dist-test-slave-qfk4"
server_key: "2ced8c21e6adc1b9c21f9c8143c3b65c"
server_key_iv: "26a354610d8c9e53ea35228ce2f77191"
server_key_version: "encryptionkey@0"
I20260619 06:40:22.758988 13866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "871d0d204466410e8683d77affe62f3a"
format_stamp: "Formatted at 2026-06-19 06:40:22 on dist-test-slave-qfk4"
server_key: "2ced8c21e6adc1b9c21f9c8143c3b65c"
server_key_iv: "26a354610d8c9e53ea35228ce2f77191"
server_key_version: "encryptionkey@0"
I20260619 06:40:22.763913 13866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 06:40:22.767395 13881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:22.768932 13866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:22.769068 13866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "871d0d204466410e8683d77affe62f3a"
format_stamp: "Formatted at 2026-06-19 06:40:22 on dist-test-slave-qfk4"
server_key: "2ced8c21e6adc1b9c21f9c8143c3b65c"
server_key_iv: "26a354610d8c9e53ea35228ce2f77191"
server_key_version: "encryptionkey@0"
I20260619 06:40:22.769230 13866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:22.789464 13866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:22.793593 13866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:22.793816 13866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:22.810775 13866 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:33983
I20260619 06:40:22.810822 13932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:33983 every 8 connection(s)
I20260619 06:40:22.812067 13866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:22.812767 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 13866
I20260619 06:40:22.812943 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:22.813627 7755 external_mini_cluster.cc:1428] Setting key 06c7a60bcc87eb93e835b6ab69e99c76
I20260619 06:40:22.816324 13933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:40:22 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:22.826673 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a: Bootstrap starting.
I20260619 06:40:22.829754 13933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:22.830726 13933 log.cc:824] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:22.833817 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a: No bootstrap required, opened a new log
I20260619 06:40:22.834805 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:22.815581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.7.146.254:33983)
0619 06:40:22.816373 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:22.816388 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:22.816437 (+ 49us) server_negotiation.cc:407] Connection header received
0619 06:40:22.818024 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:22.818045 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:22.818400 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:22.818758 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:22.820139 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:22.821620 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:22.822792 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:22.823449 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:22.823636 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:22.827535 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:22.827543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:22.827557 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:22.827598 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:22.830511 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:22.831185 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:22.831195 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:22.831204 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:22.831404 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:22.831802 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:22.831805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:22.831807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:22.832448 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:22.832761 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:22.833078 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 06:40:22.833488 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":407,"threads_started":1}
I20260619 06:40:22.841938 13933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } }
I20260619 06:40:22.842269 13933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:22.842342 13933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871d0d204466410e8683d77affe62f3a, State: Initialized, Role: FOLLOWER
I20260619 06:40:22.842814 13933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } }
I20260619 06:40:22.843015 13933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:22.843109 13933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:22.843238 13933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:22.845346 13933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } }
I20260619 06:40:22.845899 13933 leader_election.cc:302] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 871d0d204466410e8683d77affe62f3a; no voters:
I20260619 06:40:22.846524 13933 leader_election.cc:288] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:22.846612 13938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:22.847275 13938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 1 LEADER]: Becoming Leader. State: Replica: 871d0d204466410e8683d77affe62f3a, State: Running, Role: LEADER
I20260619 06:40:22.847898 13938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } }
I20260619 06:40:22.848822 13933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:22.851009 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 871d0d204466410e8683d77affe62f3a. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } }
I20260619 06:40:22.851383 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:22.851018 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } }
I20260619 06:40:22.852007 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:22.853283 13944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:22.858479 13944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:22.867203 13944 catalog_manager.cc:1347] Generated new cluster ID: eb2f1e620f71414cb0726f2011d30b04
I20260619 06:40:22.867308 13944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:22.875790 13944 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:22.877166 13944 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:22.886893 13944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a: Generated new TSK 0
I20260619 06:40:22.887950 13944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab.
Entry for principal oryx/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab.
I20260619 06:40:22.947563 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33983
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab
--principal=oryx/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:23.268394 13959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:23.268697 13959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:23.268827 13959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:23.276087 13959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:23.276201 13959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:23.276350 13959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:40:23.285969 13959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab
--principal=oryx/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:23.287232 13959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:23.288522 13959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:23.301246 13965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:23.301173 13964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:23.301219 13967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:23.301985 13959 server_base.cc:1034] running on GCE node
I20260619 06:40:23.303031 13959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:23.304008 13959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:23.305205 13959 hybrid_clock.cc:648] HybridClock initialized: now 1781851223305157 us; error 44 us; skew 500 ppm
I20260619 06:40:23.305506 13959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:23 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851223, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:23.308600 13959 init.cc:377] Logged in from keytab as oryx/127.7.146.193@KRBTEST.COM (short username oryx)
I20260619 06:40:23.309986 13959 webserver.cc:460] Webserver started at http://127.7.146.193:37763/ using document root <none> and password file <none>
I20260619 06:40:23.310626 13959 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:23.310720 13959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:23.310963 13959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:23.313776 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "8ff7fd3845374e38a40f1eda191ec548"
format_stamp: "Formatted at 2026-06-19 06:40:23 on dist-test-slave-qfk4"
server_key: "4f45d44a70e553f5254f1edb3529d0ea"
server_key_iv: "c558614ee92597d31285eb7b95bb56d5"
server_key_version: "encryptionkey@0"
I20260619 06:40:23.314468 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "8ff7fd3845374e38a40f1eda191ec548"
format_stamp: "Formatted at 2026-06-19 06:40:23 on dist-test-slave-qfk4"
server_key: "4f45d44a70e553f5254f1edb3529d0ea"
server_key_iv: "c558614ee92597d31285eb7b95bb56d5"
server_key_version: "encryptionkey@0"
I20260619 06:40:23.319574 13959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:40:23.323113 13974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:23.324537 13959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:40:23.324726 13959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "8ff7fd3845374e38a40f1eda191ec548"
format_stamp: "Formatted at 2026-06-19 06:40:23 on dist-test-slave-qfk4"
server_key: "4f45d44a70e553f5254f1edb3529d0ea"
server_key_iv: "c558614ee92597d31285eb7b95bb56d5"
server_key_version: "encryptionkey@0"
I20260619 06:40:23.324908 13959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:23.380170 13959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:23.384297 13959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:23.384536 13959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:23.385459 13959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:23.386861 13959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:23.386940 13959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:23.387053 13959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:23.387102 13959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:23.417058 13959 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:41477
I20260619 06:40:23.417136 14086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:41477 every 8 connection(s)
I20260619 06:40:23.418298 13959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:40:23.427047 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 13959
I20260619 06:40:23.427242 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:40:23.428047 7755 external_mini_cluster.cc:1428] Setting key 656ffe605acf79df0f6534f11f03fac0
Jun 19 06:40:23 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851223, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:23.439570 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:23.420874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:48685 (local address 127.7.146.254:33983)
0619 06:40:23.421367 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:23.421373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:23.422354 (+ 981us) server_negotiation.cc:407] Connection header received
0619 06:40:23.424084 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:23.424088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:23.424160 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:23.424256 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:23.425837 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:23.426642 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:23.428966 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:23.429446 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:23.429558 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:23.433172 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:23.433178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:23.433182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:23.433218 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:23.436257 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:23.437212 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:23.437217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:23.437219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:23.437346 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:23.437814 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:23.437818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:23.437820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:23.438121 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:23.438232 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:23.439127 (+ 895us) server_negotiation.cc:299] Negotiation successful
0619 06:40:23.439297 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
I20260619 06:40:23.440861 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:23.421455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.193:48685)
0619 06:40:23.422190 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:23.422228 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:23.423814 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:23.424453 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:23.424461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:23.425056 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:23.425649 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:23.425666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:23.426823 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:23.426829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:23.428815 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:23.428827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:23.429659 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:23.429666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:23.429957 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:23.430670 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:23.430691 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:23.432942 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:23.436420 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:23.436430 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:23.436447 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:23.437059 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:23.437448 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:23.437453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:23.437456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:23.437692 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:23.438237 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:23.438245 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:23.438654 (+ 409us) client_negotiation.cc:769] Sending connection context
0619 06:40:23.438947 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 06:40:23.439484 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":256,"threads_started":1}
I20260619 06:40:23.442409 14087 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33983
I20260619 06:40:23.442701 14087 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:23.443463 14087 heartbeater.cc:511] Master 127.7.146.254:33983 requested a full tablet report, sending...
I20260619 06:40:23.445478 13898 ts_manager.cc:194] Registered new tserver with Master: 8ff7fd3845374e38a40f1eda191ec548 (127.7.146.193:41477)
I20260619 06:40:23.446880 13898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:48685
WARNING: no policy specified for oryx/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab.
Entry for principal oryx/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab.
I20260619 06:40:23.471933 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33983
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab
--principal=oryx/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:23.779634 14093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:23.779937 14093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:23.780061 14093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:23.787721 14093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:23.787878 14093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:23.788045 14093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:40:23.797909 14093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab
--principal=oryx/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:23.799257 14093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:23.800597 14093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:23.813474 14098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:23.813531 14099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:23.814105 14093 server_base.cc:1034] running on GCE node
W20260619 06:40:23.814270 14101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:23.814904 14093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:23.815862 14093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:23.817065 14093 hybrid_clock.cc:648] HybridClock initialized: now 1781851223817005 us; error 53 us; skew 500 ppm
I20260619 06:40:23.817360 14093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:23 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851223, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:23.820430 14093 init.cc:377] Logged in from keytab as oryx/127.7.146.194@KRBTEST.COM (short username oryx)
I20260619 06:40:23.826121 14093 webserver.cc:460] Webserver started at http://127.7.146.194:42129/ using document root <none> and password file <none>
I20260619 06:40:23.826792 14093 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:23.826891 14093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:23.827167 14093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:23.830206 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "7022dfc23a67495c9eac0af30f5ce398"
format_stamp: "Formatted at 2026-06-19 06:40:23 on dist-test-slave-qfk4"
server_key: "8eaccfcd5255c6d69f290fd0b9aee613"
server_key_iv: "a190e333ce67dda0ee676d36554dabba"
server_key_version: "encryptionkey@0"
I20260619 06:40:23.831102 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "7022dfc23a67495c9eac0af30f5ce398"
format_stamp: "Formatted at 2026-06-19 06:40:23 on dist-test-slave-qfk4"
server_key: "8eaccfcd5255c6d69f290fd0b9aee613"
server_key_iv: "a190e333ce67dda0ee676d36554dabba"
server_key_version: "encryptionkey@0"
I20260619 06:40:23.836053 14093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 06:40:23.839692 14108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:23.841190 14093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:23.841334 14093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "7022dfc23a67495c9eac0af30f5ce398"
format_stamp: "Formatted at 2026-06-19 06:40:23 on dist-test-slave-qfk4"
server_key: "8eaccfcd5255c6d69f290fd0b9aee613"
server_key_iv: "a190e333ce67dda0ee676d36554dabba"
server_key_version: "encryptionkey@0"
I20260619 06:40:23.841527 14093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:23.875690 14093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:23.879841 14093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:23.880270 14093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:23.881425 14093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:23.882822 14093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:23.882901 14093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:23.883005 14093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:23.883064 14093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:23.913882 14093 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:33669
I20260619 06:40:23.913970 14220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:33669 every 8 connection(s)
I20260619 06:40:23.915244 14093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:40:23.920419 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 14093
I20260619 06:40:23.920621 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:40:23.921151 7755 external_mini_cluster.cc:1428] Setting key a486e5e7787fecfcb50325fa9384cc39
Jun 19 06:40:23 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851223, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:23.936106 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:23.917835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:57421 (local address 127.7.146.254:33983)
0619 06:40:23.918076 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:23.918080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:23.919273 (+ 1193us) server_negotiation.cc:407] Connection header received
0619 06:40:23.921595 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:23.921600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:23.921673 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:23.921808 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:23.923460 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:23.924234 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:23.926338 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:23.926758 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:23.926860 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:23.930728 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:23.930733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:23.930736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:23.930767 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:23.933273 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:23.933958 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:23.933962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:23.933966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:23.934103 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:23.934516 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:23.934519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:23.934521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:23.934772 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:23.934895 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:23.935563 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 06:40:23.935779 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260619 06:40:23.937129 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:23.918364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.194:57421)
0619 06:40:23.919106 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:23.919147 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:23.921273 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:23.922039 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:23.922051 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:23.922598 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:23.923283 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:23.923309 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:23.924388 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:23.924394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:23.926188 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:23.926201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:23.926974 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:23.926980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:23.927230 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:23.927932 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:23.927972 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:23.930519 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:23.933405 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:23.933414 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:23.933431 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:23.933830 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:23.934219 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:23.934223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:23.934227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:23.934404 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:23.934895 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:23.934903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:23.935220 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 06:40:23.935456 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 06:40:23.935981 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":264,"threads_started":1}
I20260619 06:40:23.938486 14221 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33983
I20260619 06:40:23.938751 14221 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.7.146.195@KRBTEST.COM; defaulting to no policy
I20260619 06:40:23.939421 14221 heartbeater.cc:511] Master 127.7.146.254:33983 requested a full tablet report, sending...
I20260619 06:40:23.940899 13898 ts_manager.cc:194] Registered new tserver with Master: 7022dfc23a67495c9eac0af30f5ce398 (127.7.146.194:33669)
I20260619 06:40:23.941957 13898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:57421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab.
Entry for principal oryx/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab.
I20260619 06:40:23.965350 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33983
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab
--principal=oryx/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:24.287621 14226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:24.287879 14226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:24.287977 14226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:24.295106 14226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:24.295218 14226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:24.295328 14226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:40:24.305205 14226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab
--principal=oryx/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:24.306398 14226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:24.307626 14226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:24.319695 14234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:24.319736 14232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:24.319691 14231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:24.320719 14226 server_base.cc:1034] running on GCE node
I20260619 06:40:24.321336 14226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:24.322252 14226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:24.323457 14226 hybrid_clock.cc:648] HybridClock initialized: now 1781851224323420 us; error 32 us; skew 500 ppm
I20260619 06:40:24.323746 14226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:24 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851224, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:24.326804 14226 init.cc:377] Logged in from keytab as oryx/127.7.146.195@KRBTEST.COM (short username oryx)
I20260619 06:40:24.328254 14226 webserver.cc:460] Webserver started at http://127.7.146.195:36123/ using document root <none> and password file <none>
I20260619 06:40:24.329003 14226 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:24.329101 14226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:24.329352 14226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:24.332144 14226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "7bf794b451df4f7a999848e2c06d0457"
format_stamp: "Formatted at 2026-06-19 06:40:24 on dist-test-slave-qfk4"
server_key: "54d8d27cf06b4409b5304068f9590630"
server_key_iv: "6411d26b101d649a0192cddd701abf97"
server_key_version: "encryptionkey@0"
I20260619 06:40:24.332862 14226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "7bf794b451df4f7a999848e2c06d0457"
format_stamp: "Formatted at 2026-06-19 06:40:24 on dist-test-slave-qfk4"
server_key: "54d8d27cf06b4409b5304068f9590630"
server_key_iv: "6411d26b101d649a0192cddd701abf97"
server_key_version: "encryptionkey@0"
I20260619 06:40:24.337642 14226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:40:24.341019 14241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:24.342358 14226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:40:24.342504 14226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "7bf794b451df4f7a999848e2c06d0457"
format_stamp: "Formatted at 2026-06-19 06:40:24 on dist-test-slave-qfk4"
server_key: "54d8d27cf06b4409b5304068f9590630"
server_key_iv: "6411d26b101d649a0192cddd701abf97"
server_key_version: "encryptionkey@0"
I20260619 06:40:24.342677 14226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:24.357163 14226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:24.361194 14226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:24.361423 14226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:24.362313 14226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:24.363621 14226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:24.363701 14226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:24.363783 14226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:24.363827 14226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:24.393846 14226 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:32911
I20260619 06:40:24.393887 14353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:32911 every 8 connection(s)
I20260619 06:40:24.395066 14226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:40:24.400091 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 14226
I20260619 06:40:24.400267 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:40:24.400815 7755 external_mini_cluster.cc:1428] Setting key 7ef2f856da416e239f1a6a42d3732c1a
Jun 19 06:40:24 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851224, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:24.415524 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:24.397677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:49003 (local address 127.7.146.254:33983)
0619 06:40:24.397868 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:24.397872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:24.398989 (+ 1117us) server_negotiation.cc:407] Connection header received
0619 06:40:24.401287 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:24.401292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:24.401383 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:24.401532 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:24.403457 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.404318 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.405923 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.406331 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.406438 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:24.409884 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:24.409890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:24.409895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:24.409933 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:24.412271 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.413054 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.413058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.413061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.413221 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.413650 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.413654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.413656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.413986 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:24.414110 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:24.415002 (+ 892us) server_negotiation.cc:299] Negotiation successful
0619 06:40:24.415231 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:40:24.416754 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:24.398162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.195:49003)
0619 06:40:24.398830 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:24.398869 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:24.400937 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:24.401786 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:24.401800 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:24.402556 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:24.403295 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:24.403316 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.404470 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.404473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:24.405778 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:24.405791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.406545 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.406551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:24.406781 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:24.407410 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:24.407433 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:24.409665 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:24.412412 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:24.412419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:24.412434 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:24.412920 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:24.413320 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:24.413323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:24.413326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:24.413538 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:24.414102 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:24.414108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:24.414569 (+ 461us) client_negotiation.cc:769] Sending connection context
0619 06:40:24.414835 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 06:40:24.415343 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":252,"threads_started":1}
I20260619 06:40:24.418141 14354 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33983
I20260619 06:40:24.418460 14354 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:24.419298 14354 heartbeater.cc:511] Master 127.7.146.254:33983 requested a full tablet report, sending...
I20260619 06:40:24.420944 13898 ts_manager.cc:194] Registered new tserver with Master: 7bf794b451df4f7a999848e2c06d0457 (127.7.146.195:32911)
I20260619 06:40:24.422024 13898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:49003
I20260619 06:40:24.429061 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.146.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.253@KRBTEST.COM" created.
I20260619 06:40:24.450033 14087 heartbeater.cc:503] Master 127.7.146.254:33983 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.253.keytab.
Entry for principal oryx/127.7.146.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.253.keytab.
I20260619 06:40:24.477731 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.253:35469
--webserver_interface=127.7.146.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.253.keytab
--principal=oryx/127.7.146.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:33983,127.7.146.253:35469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:24.791901 14359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:24.792213 14359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:24.792346 14359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:24.799705 14359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:24.799841 14359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:24.799929 14359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:24.800007 14359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:24.800065 14359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:24.810015 14359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:33983,127.7.146.253:35469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.253.keytab
--principal=oryx/127.7.146.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.253:35469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.146.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
W20260619 06:40:24.810400 14359 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.146.254:33983,127.7.146.253:35469'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 06:40:24.811431 14359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:24.812742 14359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:24.823817 14364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:24.823915 14365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:24.823974 14367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:24.824512 14359 server_base.cc:1034] running on GCE node
I20260619 06:40:24.825400 14359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:24.826853 14359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:24.828090 14359 hybrid_clock.cc:648] HybridClock initialized: now 1781851224828018 us; error 48 us; skew 500 ppm
I20260619 06:40:24.828403 14359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:24 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851224, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:24.831359 14359 init.cc:377] Logged in from keytab as oryx/127.7.146.253@KRBTEST.COM (short username oryx)
I20260619 06:40:24.832683 14359 webserver.cc:460] Webserver started at http://127.7.146.253:38253/ using document root <none> and password file <none>
I20260619 06:40:24.833336 14359 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:24.833428 14359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:24.833711 14359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:24.836602 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data/instance:
uuid: "61e84be559624a4586895c0acee1544c"
format_stamp: "Formatted at 2026-06-19 06:40:24 on dist-test-slave-qfk4"
server_key: "1458d14930134211299c670d927358c2"
server_key_iv: "140ee8c68e96a056633ccc682b6aca97"
server_key_version: "encryptionkey@0"
I20260619 06:40:24.837352 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/wal/instance:
uuid: "61e84be559624a4586895c0acee1544c"
format_stamp: "Formatted at 2026-06-19 06:40:24 on dist-test-slave-qfk4"
server_key: "1458d14930134211299c670d927358c2"
server_key_iv: "140ee8c68e96a056633ccc682b6aca97"
server_key_version: "encryptionkey@0"
I20260619 06:40:24.842594 14359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 06:40:24.846565 14374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:24.848050 14359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:40:24.848209 14359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/wal
uuid: "61e84be559624a4586895c0acee1544c"
format_stamp: "Formatted at 2026-06-19 06:40:24 on dist-test-slave-qfk4"
server_key: "1458d14930134211299c670d927358c2"
server_key_iv: "140ee8c68e96a056633ccc682b6aca97"
server_key_version: "encryptionkey@0"
I20260619 06:40:24.848420 14359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:24.873075 14359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:24.877048 14359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:24.877259 14359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:24.893782 14359 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.253:35469
I20260619 06:40:24.893800 14425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.253:35469 every 8 connection(s)
I20260619 06:40:24.895092 14359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
I20260619 06:40:24.895761 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 14359
I20260619 06:40:24.899420 14426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:24.907847 14426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:40:24 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.146.253@KRBTEST.COM
Jun 19 06:40:24 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851224, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.253@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:24.920858 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:24.898054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.7.146.253:35469)
0619 06:40:24.898825 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:24.898839 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:24.898886 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:40:24.901100 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:24.901126 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:24.901601 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:24.902030 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:24.903340 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.905504 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.906857 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.907834 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.908027 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:24.912635 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:24.912644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:24.912656 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:24.912693 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:24.916310 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.917217 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.917227 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.917235 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.917443 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.917865 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.917870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.917873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.918556 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:24.918817 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:24.919345 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 06:40:24.919759 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":360,"threads_started":1}
I20260619 06:40:24.925203 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:24.910617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.7.146.254:33983)
0619 06:40:24.910765 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:24.910769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:24.911646 (+ 877us) server_negotiation.cc:407] Connection header received
0619 06:40:24.911849 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:24.911854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:24.911930 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:24.912014 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:24.913249 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.914085 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.915734 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.916159 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.916270 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:24.919876 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:24.919880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:24.919884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:24.920019 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:24.922600 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.923285 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.923289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.923291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.923443 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.923787 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.923792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.923795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.924107 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:24.924234 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:24.924672 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 06:40:24.924875 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:40:24.925262 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:24.910681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:33524)
0619 06:40:24.911490 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:24.911539 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:24.911699 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:24.912095 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:24.912102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:24.912792 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:24.913085 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:24.913094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.914233 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.914237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:24.915603 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:24.915615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.916338 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.916345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:24.916525 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:24.917179 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:24.917219 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:24.919707 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:24.922726 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:24.922736 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:24.922744 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:24.923174 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:24.923557 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:24.923562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:24.923566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:24.923693 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:24.924230 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:24.924237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:24.924395 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 06:40:24.924559 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 06:40:24.924877 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":407,"threads_started":1}
I20260619 06:40:24.926653 14426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:40:24 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851224, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.253@KRBTEST.COM for oryx/127.7.146.253@KRBTEST.COM
I20260619 06:40:24.938262 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:24.927389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35469 (local address 127.0.0.1:57210)
0619 06:40:24.927586 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:24.927604 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:24.927691 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:24.928048 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:24.928051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:24.928365 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:24.928520 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:24.928528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.929638 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.929642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:24.930475 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:24.930487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.931143 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.931147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:24.931282 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:24.931772 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:24.931789 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:24.933701 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:24.936279 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:24.936283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:24.936285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:24.936665 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:24.937037 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:24.937040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:24.937042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:24.937156 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:24.937605 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:24.937608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:24.937722 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 06:40:24.937823 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:40:24.938031 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 06:40:24.938513 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:24.927331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57210 (local address 127.7.146.253:35469)
0619 06:40:24.927484 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:24.927489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:24.927685 (+ 196us) server_negotiation.cc:407] Connection header received
0619 06:40:24.927773 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:24.927779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:24.927912 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:24.928033 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:24.928659 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.929498 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.930605 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.931012 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.931111 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:24.933893 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:24.933906 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:24.933909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:24.933937 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:24.936166 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.936762 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.936766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.936768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.936944 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.937246 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.937250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.937252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.937520 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:24.937619 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:24.938010 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 06:40:24.938181 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:40:24.943002 14426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: Bootstrap starting.
I20260619 06:40:24.944795 14221 heartbeater.cc:503] Master 127.7.146.254:33983 was elected leader, sending a full tablet report...
I20260619 06:40:24.946431 14426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:24.947443 14426 log.cc:824] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:24.950142 14426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: No bootstrap required, opened a new log
I20260619 06:40:24.957185 14426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } }
I20260619 06:40:24.957501 14426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:24.957587 14426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e84be559624a4586895c0acee1544c, State: Initialized, Role: FOLLOWER
I20260619 06:40:24.958113 14426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } }
I20260619 06:40:24.958896 14432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } } }
I20260619 06:40:24.959206 14432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:40:24.959821 14426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:40:24.967562 14443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:40:24.967681 14443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:40:24.970108 14359 master_runner.cc:428] Detected that this master 61e84be559624a4586895c0acee1544c is joining an existing cluster
I20260619 06:40:24.970196 14359 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.146.253:35469
I20260619 06:40:24.995460 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:24.974743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.7.146.254:33983)
0619 06:40:24.974920 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:24.974925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:24.981139 (+ 6214us) server_negotiation.cc:407] Connection header received
0619 06:40:24.981298 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:24.981305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:24.981396 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:24.981487 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:24.982917 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.984043 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.986180 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.986772 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.986888 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:24.989105 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:24.989111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:24.989115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:24.989152 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:24.992476 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.993233 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.993238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.993242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.993384 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.993835 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.993840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.993843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.994175 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:24.994308 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:24.994858 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 06:40:24.995094 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:40:24.995472 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:24.973840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.7.146.253:35469)
0619 06:40:24.974048 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:24.974054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:24.981719 (+ 7665us) server_negotiation.cc:407] Connection header received
0619 06:40:24.981847 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:24.981852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:24.981924 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:24.982020 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:24.983243 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.984273 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.985643 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.986177 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.986291 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:24.989084 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:24.989090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:24.989094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:24.989128 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:24.992764 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.993480 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.993485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.993489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.993674 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:24.994043 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:24.994047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:24.994050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:24.994391 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:24.994513 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:24.994943 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 06:40:24.995125 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":44}
I20260619 06:40:24.996354 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:24.974136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35469 (local address 127.0.0.1:57224)
0619 06:40:24.981627 (+ 7491us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:24.981642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:24.981748 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:24.982169 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:24.982177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:24.982804 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:24.983108 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:24.983119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.984436 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.984442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:24.985522 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:24.985536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.986538 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.986544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:24.986706 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:24.987378 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:24.987399 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:24.988900 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:24.992912 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:24.992917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:24.992920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:24.993380 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:24.993798 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:24.993802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:24.993805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:24.993945 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:24.994576 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:24.994580 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:24.994716 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 06:40:24.995649 (+ 933us) client_negotiation.cc:240] Negotiation successful
0619 06:40:24.995923 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7388,"thread_start_us":7217,"threads_started":1}
I20260619 06:40:24.996759 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:24.973920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:33540)
0619 06:40:24.981003 (+ 7083us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:24.981027 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:24.981172 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:24.981622 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:24.981628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:24.982331 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:24.982737 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:24.982748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.984172 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.984177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:24.986022 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:24.986046 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:24.987114 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:24.987121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:24.987290 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:24.987793 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:24.987813 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:24.988900 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:24.992653 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:24.992658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:24.992663 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:24.993098 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:24.993506 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:24.993510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:24.993514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:24.993719 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:24.994380 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:24.994385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:24.994554 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 06:40:24.995714 (+ 1160us) client_negotiation.cc:240] Negotiation successful
0619 06:40:24.996104 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6944,"mutex_wait_us":113,"spinlock_wait_cycles":26880,"thread_start_us":6803,"threads_started":1}
WARNING: no policy specified for oryx/127.7.146.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.252@KRBTEST.COM" created.
Jun 19 06:40:25 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851222, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.254@KRBTEST.COM for oryx/127.7.146.253@KRBTEST.COM
I20260619 06:40:25.019656 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.003022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.7.146.253:35469)
0619 06:40:25.003188 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.003194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.003890 (+ 696us) server_negotiation.cc:407] Connection header received
0619 06:40:25.004419 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.004425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.004518 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.004641 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.006118 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.007229 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.009078 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.009774 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.009910 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.013343 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.013348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.013351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.013384 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.016792 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.017589 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.017593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.017596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.017728 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.018092 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.018095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.018098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.018401 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.018543 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.019029 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.019252 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:40:25.019685 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.003092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35469 (local address 127.0.0.1:57234)
0619 06:40:25.003760 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:25.003803 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:25.004033 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:25.004860 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:25.004868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:25.005682 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:25.005938 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.005951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.007366 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.007371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.008867 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.008890 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.009953 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.010026 (+ 73us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.010261 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.010919 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:25.010938 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:25.013142 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:25.016945 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.016962 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.016972 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.017453 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.017842 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.017847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.017851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.017983 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.018543 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:25.018556 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:25.018733 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 06:40:25.018917 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 06:40:25.019252 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":264,"threads_started":1}
I20260619 06:40:25.021988 13898 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.146.253:35469: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "61e84be559624a4586895c0acee1544c"
member_type: NON_VOTER
last_known_addr {
host: "127.7.146.253"
port: 35469
}
attrs {
promote: true
}
}
dest_uuid: "871d0d204466410e8683d77affe62f3a"
cas_config_opid_index: -1
I20260619 06:40:25.022926 13898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } }
I20260619 06:40:25.025701 14456 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 61e84be559624a4586895c0acee1544c (127.7.146.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } } }
I20260619 06:40:25.027122 14401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:25.029137 14456 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.146.253:35469
I20260619 06:40:25.029524 14456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } } }
I20260619 06:40:25.029676 14456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:25.031543 14359 master.cc:537] Master@127.7.146.253:35469 shutting down...
I20260619 06:40:25.031814 14401 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 FOLLOWER]: Refusing update from remote peer 871d0d204466410e8683d77affe62f3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 06:40:25.033056 14455 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.252.keytab.
Entry for principal oryx/127.7.146.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.252.keytab.
W20260619 06:40:25.036190 13884 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 06:40:25.036679 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.252:33227
--webserver_interface=127.7.146.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.252.keytab
--principal=oryx/127.7.146.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:33983,127.7.146.253:35469,127.7.146.252:33227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:25.044902 13884 consensus_peers.cc:487] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a -> Peer 61e84be559624a4586895c0acee1544c (127.7.146.253:35469): Couldn't send request to peer 61e84be559624a4586895c0acee1544c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 06:40:25.047066 14359 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:25.047576 14359 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:25.047781 14359 tablet_replica.cc:330] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: stopping tablet replica
I20260619 06:40:25.053828 14359 master.cc:552] Master@127.7.146.253:35469 shutdown complete.
I20260619 06:40:25.053902 14359 master_runner.cc:305] Clearing existing system tablet
I20260619 06:40:25.054301 14359 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:25.054375 14359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:25.056653 14359 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 06:40:25.058053 14461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:25.058326 14359 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:25.058573 14359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:40:25.058650 14359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/wal
uuid: "61e84be559624a4586895c0acee1544c"
format_stamp: "Formatted at 2026-06-19 06:40:24 on dist-test-slave-qfk4"
server_key: "1458d14930134211299c670d927358c2"
server_key_iv: "140ee8c68e96a056633ccc682b6aca97"
server_key_version: "encryptionkey@0"
I20260619 06:40:25.062278 14359 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:25.066895 14359 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:40:25.067057 14359 log.cc:1196] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 06:40:25.067438 14359 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: Deleting consensus metadata
I20260619 06:40:25.067649 14359 master_runner.cc:315] Copying system tablet from 127.7.146.254:33983
I20260619 06:40:25.069957 14359 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: tablet copy: Beginning tablet copy session from remote peer at address 127.7.146.254:33983
I20260619 06:40:25.082026 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.070688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33550 (local address 127.7.146.254:33983)
0619 06:40:25.070875 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.070882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.071385 (+ 503us) server_negotiation.cc:407] Connection header received
0619 06:40:25.071544 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.071549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.071624 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.071705 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.072643 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.073551 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.074830 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.075333 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.075443 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.077134 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.077140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.077145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.077175 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.079585 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.080158 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.080164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.080168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.080279 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.080671 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.080676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.080680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.080937 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.081064 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.081455 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.081693 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:40:25.082079 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.070653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:33550)
0619 06:40:25.071262 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:25.071277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:25.071419 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:25.071747 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:25.071752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:25.072272 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:25.072480 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.072492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.073691 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.073697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.074678 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.074692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.075470 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.075476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.075651 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.076212 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:25.076229 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:25.076947 (+ 718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:25.079704 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.079710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.079714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.080054 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.080373 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.080378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.080382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.080504 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.081051 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:25.081056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:25.081225 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 06:40:25.081352 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 06:40:25.081648 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":347,"threads_started":1}
I20260619 06:40:25.083065 13918 tablet_copy_service.cc:140] P 871d0d204466410e8683d77affe62f3a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 61e84be559624a4586895c0acee1544c ({username='oryx', principal='oryx/127.7.146.253@KRBTEST.COM'} at 127.0.0.1:33550)
I20260619 06:40:25.083333 13918 tablet_copy_service.cc:161] P 871d0d204466410e8683d77affe62f3a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 61e84be559624a4586895c0acee1544c at {username='oryx', principal='oryx/127.7.146.253@KRBTEST.COM'} at 127.0.0.1:33550: session id = 61e84be559624a4586895c0acee1544c-00000000000000000000000000000000
I20260619 06:40:25.086360 13918 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:40:25.088217 14359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:40:25.092860 14359 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: tablet copy: Starting download of 0 data blocks...
I20260619 06:40:25.093155 14359 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: tablet copy: Starting download of 1 WAL segments...
I20260619 06:40:25.095551 14359 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:40:25.098752 13918 tablet_copy_service.cc:342] P 871d0d204466410e8683d77affe62f3a: Request end of tablet copy session 61e84be559624a4586895c0acee1544c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.146.253@KRBTEST.COM'} at 127.0.0.1:33550
I20260619 06:40:25.098968 13918 tablet_copy_service.cc:433] P 871d0d204466410e8683d77affe62f3a: ending tablet copy session 61e84be559624a4586895c0acee1544c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 61e84be559624a4586895c0acee1544c
I20260619 06:40:25.110488 14359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:25.113186 14468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:25.114457 14469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:25.122133 14471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:25.126014 14359 server_base.cc:1034] running on GCE node
I20260619 06:40:25.126377 14359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:25.127067 14359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:25.128275 14359 hybrid_clock.cc:648] HybridClock initialized: now 1781851225128215 us; error 35 us; skew 500 ppm
I20260619 06:40:25.128527 14359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:25 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851225, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:25.130676 14359 init.cc:377] Logged in from keytab as oryx/127.7.146.253@KRBTEST.COM (short username oryx)
I20260619 06:40:25.133529 14359 webserver.cc:460] Webserver started at http://127.7.146.253:32921/ using document root <none> and password file <none>
I20260619 06:40:25.133869 14359 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:25.133958 14359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:25.137037 14359 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:40:25.139147 14478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:25.139542 14359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 06:40:25.139727 14359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/wal
uuid: "61e84be559624a4586895c0acee1544c"
format_stamp: "Formatted at 2026-06-19 06:40:24 on dist-test-slave-qfk4"
server_key: "1458d14930134211299c670d927358c2"
server_key_iv: "140ee8c68e96a056633ccc682b6aca97"
server_key_version: "encryptionkey@0"
I20260619 06:40:25.139967 14359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:25.168326 14359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:25.172453 14359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:25.200623 14359 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.253:35469
I20260619 06:40:25.200677 14529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.253:35469 every 8 connection(s)
I20260619 06:40:25.201635 14359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
I20260619 06:40:25.206799 14530 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:40:25.210590 14530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: Bootstrap starting.
I20260619 06:40:25.232345 14530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:40:25.232867 14530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: Bootstrap complete.
I20260619 06:40:25.233966 14530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } }
I20260619 06:40:25.234263 14530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e84be559624a4586895c0acee1544c, State: Initialized, Role: LEARNER
I20260619 06:40:25.234543 14530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } }
I20260619 06:40:25.235396 14533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } } }
I20260619 06:40:25.235700 14530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:25.235797 14533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: This master's current role is: LEARNER
I20260619 06:40:25.244114 14544 catalog_manager.cc:1259] Loaded cluster ID: eb2f1e620f71414cb0726f2011d30b04
I20260619 06:40:25.244225 14544 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: loading cluster ID for follower catalog manager: success
I20260619 06:40:25.247692 14544 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: acquiring CA information for follower catalog manager: success
I20260619 06:40:25.250172 14544 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:40:25.411522 14459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:25.411805 14459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:25.411938 14459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:25.420459 14459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:25.420611 14459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:25.420742 14459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:25.420835 14459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:25.420899 14459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:25.424553 14354 heartbeater.cc:503] Master 127.7.146.254:33983 was elected leader, sending a full tablet report...
I20260619 06:40:25.431412 14459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:33983,127.7.146.253:35469,127.7.146.252:33227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.252.keytab
--principal=oryx/127.7.146.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.252:33227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.146.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:25.432750 14459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:25.433965 14459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:25.445118 14550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:25.445202 14549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:25.445178 14552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:25.446100 14459 server_base.cc:1034] running on GCE node
I20260619 06:40:25.446861 14459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:25.448276 14459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:25.449496 14459 hybrid_clock.cc:648] HybridClock initialized: now 1781851225449443 us; error 43 us; skew 500 ppm
I20260619 06:40:25.449849 14459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:25 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851225, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:25.453106 14459 init.cc:377] Logged in from keytab as oryx/127.7.146.252@KRBTEST.COM (short username oryx)
I20260619 06:40:25.455183 14459 webserver.cc:460] Webserver started at http://127.7.146.252:39451/ using document root <none> and password file <none>
I20260619 06:40:25.455855 14459 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:25.455966 14459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:25.456226 14459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:25.459381 14459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data/instance:
uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6"
format_stamp: "Formatted at 2026-06-19 06:40:25 on dist-test-slave-qfk4"
server_key: "43ef4e1bd715f51b44af697806e674db"
server_key_iv: "34736b3d90c07f3d3381b42f3b38912b"
server_key_version: "encryptionkey@0"
I20260619 06:40:25.460139 14459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/wal/instance:
uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6"
format_stamp: "Formatted at 2026-06-19 06:40:25 on dist-test-slave-qfk4"
server_key: "43ef4e1bd715f51b44af697806e674db"
server_key_iv: "34736b3d90c07f3d3381b42f3b38912b"
server_key_version: "encryptionkey@0"
I20260619 06:40:25.465477 14459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:40:25.468936 14559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:25.470351 14459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:40:25.470515 14459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/wal
uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6"
format_stamp: "Formatted at 2026-06-19 06:40:25 on dist-test-slave-qfk4"
server_key: "43ef4e1bd715f51b44af697806e674db"
server_key_iv: "34736b3d90c07f3d3381b42f3b38912b"
server_key_version: "encryptionkey@0"
I20260619 06:40:25.470700 14459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:25.484198 14459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:25.488112 14459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:25.488358 14459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:25.504630 14459 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.252:33227
I20260619 06:40:25.504652 14610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.252:33227 every 8 connection(s)
I20260619 06:40:25.506013 14459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
I20260619 06:40:25.509577 14611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:25.514082 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 14459
I20260619 06:40:25.516023 14611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:40:25 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851225, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.252@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
Jun 19 06:40:25 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.146.252@KRBTEST.COM
I20260619 06:40:25.532958 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.519092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.7.146.254:33983)
0619 06:40:25.519276 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.519282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.519812 (+ 530us) server_negotiation.cc:407] Connection header received
0619 06:40:25.520097 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.520105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.520162 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.520239 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.521418 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.522162 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.523601 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.523994 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.524107 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.527348 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.527353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.527356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.527394 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.530079 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.530772 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.530776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.530779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.530898 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.531293 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.531298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.531301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.531573 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.531695 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.532378 (+ 683us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.532632 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:40:25.534075 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.518947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:33554)
0619 06:40:25.519696 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:25.519732 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:25.519922 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:25.520312 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:25.520318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:25.520935 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:25.521269 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.521284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.522290 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.522294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.523469 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.523480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.524208 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.524215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.524339 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.524994 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:25.525023 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:25.527178 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:25.530201 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.530212 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.530225 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.530664 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.531002 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.531007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.531014 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.531194 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.531695 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:25.531704 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:25.532038 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 06:40:25.532285 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 06:40:25.532823 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":5,"thread_start_us":365,"threads_started":1}
I20260619 06:40:25.534094 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.516037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.7.146.252:33227)
0619 06:40:25.516811 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.516826 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.516874 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:40:25.519003 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.519034 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.519464 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.519780 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.521313 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.522419 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.523617 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.524125 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.524293 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.528419 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.528427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.528446 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.528477 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.531158 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.531782 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.531790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.531792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.531912 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.532235 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.532239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.532243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.532558 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.532910 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.533426 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.533643 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":534,"thread_start_us":305,"threads_started":1}
I20260619 06:40:25.535531 14611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:40:25 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851225, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.252@KRBTEST.COM for oryx/127.7.146.253@KRBTEST.COM
I20260619 06:40:25.548986 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.536289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35469 (local address 127.0.0.1:57236)
0619 06:40:25.537326 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:25.537341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:25.537439 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:25.537740 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:25.537744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:25.538118 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:25.538244 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.538251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.539600 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.539605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.540558 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.540629 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.541329 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.541334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.541505 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.542057 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:25.542073 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:25.543925 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:25.546797 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.546802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.546807 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.547155 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.547498 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.547503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.547507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.547635 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.548099 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:25.548104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:25.548241 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 06:40:25.548365 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 06:40:25.548676 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 06:40:25.549018 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.536288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.7.146.253:35469)
0619 06:40:25.536911 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.536919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.537411 (+ 492us) server_negotiation.cc:407] Connection header received
0619 06:40:25.537555 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.537560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.537626 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.537710 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.538377 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.539475 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.540736 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.541190 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.541296 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.544144 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.544149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.544153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.544183 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.546686 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.547260 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.547265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.547269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.547401 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.547728 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.547734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.547737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.547983 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.548123 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.548473 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.548689 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":311,"threads_started":1}
I20260619 06:40:25.550091 14611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:40:25 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851225, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.252@KRBTEST.COM for oryx/127.7.146.252@KRBTEST.COM
I20260619 06:40:25.561429 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.550670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33227 (local address 127.0.0.1:59776)
0619 06:40:25.550819 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:25.550840 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:25.550983 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:25.551215 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:25.551219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:25.551527 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:25.551652 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.551659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.552613 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.552617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.553478 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.553490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.554195 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.554199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.554346 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.554820 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:25.554836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:25.556715 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:25.559399 (+ 2684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.559403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.559405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.559738 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.560067 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.560070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.560073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.560206 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.560694 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:25.560697 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:25.560804 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 06:40:25.560970 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 06:40:25.561144 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 06:40:25.561486 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.550848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.7.146.252:33227)
0619 06:40:25.551016 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.551020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.551034 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:40:25.551069 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.551072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.551123 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.551194 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.551764 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.552472 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.553612 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.554042 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.554177 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.556863 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.556868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.556871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.556897 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.559275 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.559831 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.559835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.559837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.559984 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.560293 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.560297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.560299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.560544 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.560707 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.561051 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.561249 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":45}
I20260619 06:40:25.565975 14611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: Bootstrap starting.
I20260619 06:40:25.569023 14611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:25.569983 14611 log.cc:824] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:25.572521 14611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: No bootstrap required, opened a new log
I20260619 06:40:25.579602 14611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } }
I20260619 06:40:25.579944 14611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:25.580053 14611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354e9743e7cb4872ac5c3b3fe6e5f8b6, State: Initialized, Role: FOLLOWER
I20260619 06:40:25.580675 14611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } }
I20260619 06:40:25.581507 14618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } } }
I20260619 06:40:25.581779 14618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:40:25.582450 14611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:25.588617 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 13959
W20260619 06:40:25.590935 14629 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:40:25.591068 14629 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:40:25.594652 14459 master_runner.cc:428] Detected that this master 354e9743e7cb4872ac5c3b3fe6e5f8b6 is joining an existing cluster
I20260619 06:40:25.594766 14459 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.146.252:33227
Jun 19 06:40:25 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.146.193@KRBTEST.COM
I20260619 06:40:25.611141 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.590300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.7.146.193:41477)
0619 06:40:25.591609 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.591617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.591646 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:40:25.591709 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.591718 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.591955 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.592113 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.593740 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.595239 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.597097 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.597976 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.598104 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.603393 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.603408 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.603424 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.603488 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.607091 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.607946 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.607955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.607972 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.608119 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.608609 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.608614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.608618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.609046 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.609332 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.610455 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.610734 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":977,"threads_started":1}
I20260619 06:40:25.612783 14082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:25.617508 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.602784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:33566)
0619 06:40:25.603409 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:25.603426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:25.603539 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:25.604066 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:25.604070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:25.604750 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:25.605155 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.605167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.606736 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.606742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.608335 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.608352 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.609466 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.609473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.609646 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.610301 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:25.610323 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:25.611604 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:25.614497 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.614502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.614506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.614988 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.615370 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.615375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.615378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.615626 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.616281 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:25.616286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:25.616510 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 06:40:25.616749 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 06:40:25.617038 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":746,"threads_started":1}
I20260619 06:40:25.618595 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.603315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.7.146.254:33983)
0619 06:40:25.603495 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.603500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.603519 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:40:25.603704 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.603710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.603795 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.603858 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.605453 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.606552 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.608488 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.609126 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.609211 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.611782 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.611787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.611791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.611829 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.614378 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.615099 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.615105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.615109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.615246 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.615756 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.615761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.615764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.616094 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.616225 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.618050 (+ 1825us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.618266 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:40:25.633082 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.604208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.7.146.253:35469)
0619 06:40:25.604372 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.604378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.618039 (+ 13661us) server_negotiation.cc:407] Connection header received
0619 06:40:25.618109 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.618114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.618212 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.618303 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.619664 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.620781 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.622798 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.623483 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.623580 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.626193 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.626200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.626205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.626246 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.629493 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.630328 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.630333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.630337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.630487 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.630929 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.630934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.630938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.631269 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.631380 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.632437 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.632690 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:40:25.633991 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.604734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35469 (local address 127.0.0.1:57242)
0619 06:40:25.617736 (+ 13002us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:25.617754 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:25.617849 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:25.618469 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:25.618473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:25.619268 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:25.619477 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.619489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.620967 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.620973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.622637 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.622651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.623808 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.623814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.624031 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.624852 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:25.624878 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:25.625987 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:25.629668 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.629673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.629677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.630195 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.630629 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.630634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.630637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.630795 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.631470 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:25.631475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:25.631621 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 06:40:25.633328 (+ 1707us) client_negotiation.cc:240] Negotiation successful
0619 06:40:25.633615 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12909,"thread_start_us":12467,"threads_started":1}
I20260619 06:40:25.650394 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.617694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59788 (local address 127.7.146.252:33227)
0619 06:40:25.617987 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.617992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.632260 (+ 14268us) server_negotiation.cc:407] Connection header received
0619 06:40:25.635568 (+ 3308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.635574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.635696 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.635762 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.637314 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.638372 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.640365 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.641086 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.641206 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.644014 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.644020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.644025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.644072 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.647597 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.648302 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.648307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.648311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.648443 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.648846 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.648850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.648853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.649249 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.649373 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.649794 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.650019 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":184}
I20260619 06:40:25.651350 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.624483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33227 (local address 127.0.0.1:59788)
0619 06:40:25.632177 (+ 7694us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:25.632197 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:25.635366 (+ 3169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:25.636011 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:25.636016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:25.636740 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:25.637179 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.637191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.638595 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.638600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.640225 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.640239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.641521 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.641527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.641678 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.642336 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:25.642361 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:25.643830 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:25.647735 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.647740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.647744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.648194 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.648585 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.648590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.648593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.648719 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.649421 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:25.649425 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:25.649569 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 06:40:25.650650 (+ 1081us) client_negotiation.cc:240] Negotiation successful
0619 06:40:25.650943 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7441}
Jun 19 06:40:25 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851222, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.254@KRBTEST.COM for oryx/127.7.146.252@KRBTEST.COM
I20260619 06:40:25.668174 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.639917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33227 (local address 127.0.0.1:59790)
0619 06:40:25.647454 (+ 7537us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:25.647473 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:25.648679 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:25.649403 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:25.649419 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:25.652914 (+ 3495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:25.653144 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.653154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.654844 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.654850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.656301 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.656317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.657556 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.657563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.657729 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.658535 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:25.658559 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:25.661252 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:25.665304 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.665310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.665314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.665855 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.666299 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.666304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.666308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.666459 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.667160 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:25.667166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:25.667315 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 06:40:25.667448 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 06:40:25.667735 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7382,"thread_start_us":7232,"threads_started":1}
I20260619 06:40:25.669301 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.633810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59790 (local address 127.7.146.252:33227)
0619 06:40:25.641449 (+ 7639us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.641454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.647973 (+ 6519us) server_negotiation.cc:407] Connection header received
0619 06:40:25.648897 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.648902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.648984 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.649058 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.653347 (+ 4289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.654674 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.656462 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.657211 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.657301 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.661508 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.661515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.661519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.661555 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.665122 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.666014 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.666019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.666023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.666153 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.666579 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.666584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.666588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.666958 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.667079 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.668697 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.668896 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":7510,"thread_start_us":7378,"threads_started":1}
I20260619 06:40:25.670694 13898 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.146.252:33227: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6"
member_type: NON_VOTER
last_known_addr {
host: "127.7.146.252"
port: 33227
}
attrs {
promote: true
}
}
dest_uuid: "871d0d204466410e8683d77affe62f3a"
cas_config_opid_index: 5
I20260619 06:40:25.671376 13898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } }
I20260619 06:40:25.673214 14639 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 354e9743e7cb4872ac5c3b3fe6e5f8b6 (127.7.146.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:25.676879 14639 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.146.252:33227
I20260619 06:40:25.677004 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.647758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.7.146.253:35469)
0619 06:40:25.647937 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.647942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.668519 (+ 20577us) server_negotiation.cc:407] Connection header received
0619 06:40:25.669975 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.669982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.670078 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.670161 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:25.670760 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.671829 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.674277 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.675581 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.675691 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.675937 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.676587 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.676740 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:40:25.677287 14639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:25.677603 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.647654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35469 (local address 127.0.0.1:57254)
0619 06:40:25.668422 (+ 20768us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:25.668442 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:25.669781 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:25.670376 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:25.670381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:25.670406 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:25.670594 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.670605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.672030 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.672035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.674091 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.674105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.676123 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.676130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.676357 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.676416 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 06:40:25.677171 (+ 755us) client_negotiation.cc:240] Negotiation successful
0619 06:40:25.677323 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20659,"thread_start_us":21019,"threads_started":1}
I20260619 06:40:25.677628 14639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:25.679847 14459 master.cc:537] Master@127.7.146.252:33227 shutting down...
I20260619 06:40:25.681440 14586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:25.681640 14504 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 LEARNER]: Refusing update from remote peer 871d0d204466410e8683d77affe62f3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 06:40:25.682253 14505 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 06:40:25.682895 14533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 871d0d204466410e8683d77affe62f3a. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } } }
I20260619 06:40:25.683153 14533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: This master's current role is: LEARNER
I20260619 06:40:25.683926 14643 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 06:40:25.686765 14505 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 354e9743e7cb4872ac5c3b3fe6e5f8b6 (127.7.146.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:25.689780 14533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: NON_VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: true } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:25.690068 14533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: This master's current role is: LEARNER
I20260619 06:40:25.690467 14643 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a: attempting to promote NON_VOTER 61e84be559624a4586895c0acee1544c to VOTER
I20260619 06:40:25.691115 14643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } }
I20260619 06:40:25.693176 14505 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 LEARNER]: Refusing update from remote peer 871d0d204466410e8683d77affe62f3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 06:40:25.693645 14643 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 06:40:25.695623 14639 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 61e84be559624a4586895c0acee1544c (127.7.146.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:25.696211 14505 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 61e84be559624a4586895c0acee1544c (127.7.146.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:25.699640 14533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:25.699880 14533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:40:25.708527 14643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:25.708891 14643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:25.711902 14639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:25.712405 14639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:25.712747 14586 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 FOLLOWER]: Refusing update from remote peer 871d0d204466410e8683d77affe62f3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260619 06:40:25.713344 13884 consensus_peers.cc:487] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a -> Peer 354e9743e7cb4872ac5c3b3fe6e5f8b6 (127.7.146.252:33227): Couldn't send request to peer 354e9743e7cb4872ac5c3b3fe6e5f8b6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 06:40:25.715309 14459 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:25.715662 14459 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:25.715848 14459 tablet_replica.cc:330] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: stopping tablet replica
I20260619 06:40:25.721724 14459 master.cc:552] Master@127.7.146.252:33227 shutdown complete.
I20260619 06:40:25.721796 14459 master_runner.cc:305] Clearing existing system tablet
I20260619 06:40:25.722121 14459 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:25.722203 14459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:25.724305 14459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 06:40:25.725622 14650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:25.725905 14459 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:25.726159 14459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:40:25.726254 14459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/wal
uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6"
format_stamp: "Formatted at 2026-06-19 06:40:25 on dist-test-slave-qfk4"
server_key: "43ef4e1bd715f51b44af697806e674db"
server_key_iv: "34736b3d90c07f3d3381b42f3b38912b"
server_key_version: "encryptionkey@0"
I20260619 06:40:25.729471 14459 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:25.733842 14459 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:40:25.733973 14459 log.cc:1196] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 06:40:25.734368 14459 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: Deleting consensus metadata
I20260619 06:40:25.734576 14459 master_runner.cc:315] Copying system tablet from 127.7.146.254:33983
I20260619 06:40:25.736835 14459 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: tablet copy: Beginning tablet copy session from remote peer at address 127.7.146.254:33983
I20260619 06:40:25.752214 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.737556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.7.146.254:33983)
0619 06:40:25.737802 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:25.737809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:25.741168 (+ 3359us) server_negotiation.cc:407] Connection header received
0619 06:40:25.741356 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:25.741362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:25.741457 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:25.741554 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:25.742446 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.743253 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.744589 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.744999 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.745109 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.746932 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:25.746938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:25.746943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:25.746994 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:25.749704 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.750377 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.750382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.750386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.750516 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:25.750855 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:25.750860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:25.750863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:25.751123 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:25.751245 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:25.751618 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 06:40:25.751835 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 06:40:25.752238 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:25.737487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:33572)
0619 06:40:25.741041 (+ 3554us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:25.741062 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:25.741223 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:25.741624 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:25.741629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:25.742050 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:25.742263 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.742276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.743408 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.743411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.744433 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:25.744445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:25.745144 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:25.745150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:25.745413 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:25.746019 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:25.746046 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:25.746746 (+ 700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:25.749856 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.749863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.749867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.750279 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.750614 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:25.750619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:25.750623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:25.750762 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:25.751227 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:25.751232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:25.751380 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 06:40:25.751502 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 06:40:25.751811 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3350,"thread_start_us":3228,"threads_started":1}
I20260619 06:40:25.753170 13918 tablet_copy_service.cc:140] P 871d0d204466410e8683d77affe62f3a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 354e9743e7cb4872ac5c3b3fe6e5f8b6 ({username='oryx', principal='oryx/127.7.146.252@KRBTEST.COM'} at 127.0.0.1:33572)
I20260619 06:40:25.753320 13918 tablet_copy_service.cc:161] P 871d0d204466410e8683d77affe62f3a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 354e9743e7cb4872ac5c3b3fe6e5f8b6 at {username='oryx', principal='oryx/127.7.146.252@KRBTEST.COM'} at 127.0.0.1:33572: session id = 354e9743e7cb4872ac5c3b3fe6e5f8b6-00000000000000000000000000000000
I20260619 06:40:25.755690 13918 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:40:25.757233 14459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:40:25.761806 14459 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: tablet copy: Starting download of 0 data blocks...
I20260619 06:40:25.762050 14459 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: tablet copy: Starting download of 1 WAL segments...
I20260619 06:40:25.763980 14459 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:40:25.767288 13918 tablet_copy_service.cc:342] P 871d0d204466410e8683d77affe62f3a: Request end of tablet copy session 354e9743e7cb4872ac5c3b3fe6e5f8b6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.146.252@KRBTEST.COM'} at 127.0.0.1:33572
I20260619 06:40:25.767413 13918 tablet_copy_service.cc:433] P 871d0d204466410e8683d77affe62f3a: ending tablet copy session 354e9743e7cb4872ac5c3b3fe6e5f8b6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 354e9743e7cb4872ac5c3b3fe6e5f8b6
I20260619 06:40:25.777979 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 13959
I20260619 06:40:25.778404 14459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:25.788990 14657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:25.790637 14658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:25.797627 14660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:25.798418 14459 server_base.cc:1034] running on GCE node
I20260619 06:40:25.798794 14459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:25.799089 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:41477
--local_ip_for_outbound_sockets=127.7.146.193
--tserver_master_addrs=127.7.146.254:33983
--webserver_port=37763
--webserver_interface=127.7.146.193
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab
--principal=oryx/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.146.254:33983,127.7.146.253:35469
--tserver_master_addrs=127.7.146.254:33983,127.7.146.253:35469,127.7.146.252:33227 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:40:25.799650 14459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:25.800832 14459 hybrid_clock.cc:648] HybridClock initialized: now 1781851225800797 us; error 28 us; skew 500 ppm
I20260619 06:40:25.801086 14459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:25 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851225, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:25.803051 14459 init.cc:377] Logged in from keytab as oryx/127.7.146.252@KRBTEST.COM (short username oryx)
I20260619 06:40:25.809165 14459 webserver.cc:460] Webserver started at http://127.7.146.252:44043/ using document root <none> and password file <none>
I20260619 06:40:25.809480 14459 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:25.809568 14459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:25.833235 14459 fs_manager.cc:709] Time spent opening directory manager: real 0.023s user 0.011s sys 0.012s
I20260619 06:40:25.835520 14668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:25.835997 14459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:40:25.836122 14459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/wal
uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6"
format_stamp: "Formatted at 2026-06-19 06:40:25 on dist-test-slave-qfk4"
server_key: "43ef4e1bd715f51b44af697806e674db"
server_key_iv: "34736b3d90c07f3d3381b42f3b38912b"
server_key_version: "encryptionkey@0"
I20260619 06:40:25.836287 14459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:25.847087 14459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:25.850353 14459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:25.877580 14459 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.252:33227
I20260619 06:40:25.877619 14719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.252:33227 every 8 connection(s)
I20260619 06:40:25.878444 14459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
I20260619 06:40:25.883814 14720 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:40:25.886906 14720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: Bootstrap starting.
I20260619 06:40:25.899875 14720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:40:25.900303 14720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: Bootstrap complete.
I20260619 06:40:25.901160 14720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } }
I20260619 06:40:25.901381 14720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 354e9743e7cb4872ac5c3b3fe6e5f8b6, State: Initialized, Role: LEARNER
I20260619 06:40:25.901597 14720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } }
I20260619 06:40:25.902276 14723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:25.902575 14720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:25.902734 14723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:40:25.909821 14734 catalog_manager.cc:1259] Loaded cluster ID: eb2f1e620f71414cb0726f2011d30b04
I20260619 06:40:25.909900 14734 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: loading cluster ID for follower catalog manager: success
I20260619 06:40:25.912210 14734 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: acquiring CA information for follower catalog manager: success
I20260619 06:40:25.914067 14734 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:40:26.182504 14664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:26.182792 14664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:26.182941 14664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 06:40:26.192019 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.184986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.7.146.252:33227)
0619 06:40:26.185563 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:26.185569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:26.185593 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:40:26.185666 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:26.185672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:26.185755 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:26.185884 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:26.186281 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.187742 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.189320 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.190580 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.190701 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.190938 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:26.191527 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 06:40:26.191694 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":286,"threads_started":1}
I20260619 06:40:26.192068 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.184808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33227 (local address 127.0.0.1:59794)
0619 06:40:26.185282 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:26.185306 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:26.185447 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:26.185966 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:26.185969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:26.185993 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:26.186144 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.186151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.187916 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.187921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.189177 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.189191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.191137 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.191144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.191329 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.191365 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:40:26.191481 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 06:40:26.191627 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
W20260619 06:40:26.192361 14664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:26.192484 14664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:26.192680 14664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:40:26.193387 14695 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 06:40:26.193996 14723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 871d0d204466410e8683d77affe62f3a. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:26.194278 14723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:40:26.203742 14664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab
--principal=oryx/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:41477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=37763
--tserver_master_addrs=127.7.146.254:33983,127.7.146.253:35469,127.7.146.252:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:26.205281 14664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:26.206665 14664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:26.220005 14744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:26.220038 14742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:26.220060 14741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:26.221221 14664 server_base.cc:1034] running on GCE node
I20260619 06:40:26.221819 14664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:26.222762 14664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:26.223959 14664 hybrid_clock.cc:648] HybridClock initialized: now 1781851226223904 us; error 43 us; skew 500 ppm
I20260619 06:40:26.224256 14664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:26 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851226, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:26.227233 14664 init.cc:377] Logged in from keytab as oryx/127.7.146.193@KRBTEST.COM (short username oryx)
I20260619 06:40:26.228516 14664 webserver.cc:460] Webserver started at http://127.7.146.193:37763/ using document root <none> and password file <none>
I20260619 06:40:26.229238 14664 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:26.229350 14664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:26.234504 14664 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 06:40:26.237243 14751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:26.238713 14664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:26.238847 14664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "8ff7fd3845374e38a40f1eda191ec548"
format_stamp: "Formatted at 2026-06-19 06:40:23 on dist-test-slave-qfk4"
server_key: "4f45d44a70e553f5254f1edb3529d0ea"
server_key_iv: "c558614ee92597d31285eb7b95bb56d5"
server_key_version: "encryptionkey@0"
I20260619 06:40:26.239835 14664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:26.257341 14664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:26.261435 14664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:26.261659 14664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:26.262558 14664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:26.263840 14664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:26.263924 14664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:26.264000 14664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:26.264051 14664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:26.294328 14664 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:41477
I20260619 06:40:26.294384 14863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:41477 every 8 connection(s)
I20260619 06:40:26.295881 14664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:40:26.304358 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 14664
I20260619 06:40:26.304730 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 14093
Jun 19 06:40:26 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851226, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.252@KRBTEST.COM
Jun 19 06:40:26 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851226, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:26.335917 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.299602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:58149 (local address 127.7.146.252:33227)
0619 06:40:26.299811 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:26.299818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:26.301579 (+ 1761us) server_negotiation.cc:407] Connection header received
0619 06:40:26.305051 (+ 3472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:26.305056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:26.305142 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:26.305249 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:26.314950 (+ 9701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.316135 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.318794 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.319498 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.319598 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.326718 (+ 7120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:26.326727 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:26.326732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:26.326781 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:26.332040 (+ 5259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.333078 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.333084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.333088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.333234 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.333791 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.333796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.333800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.334153 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:26.334285 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:26.335310 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0619 06:40:26.335583 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 06:40:26 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.146.194@KRBTEST.COM
I20260619 06:40:26.339075 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.299034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.193:36009)
0619 06:40:26.303052 (+ 4018us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:26.303072 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:26.310013 (+ 6941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:26.316009 (+ 5996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:26.316022 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:26.316700 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:26.317002 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.317013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.318394 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.318400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.320092 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.320107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.323301 (+ 3194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.323309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.323528 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.325587 (+ 2059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:26.325612 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:26.329653 (+ 4041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:26.333646 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.333650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.333654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.334177 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.335516 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.335521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.335525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.335661 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.336764 (+ 1103us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:26.336769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:26.336909 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 06:40:26.337042 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 06:40:26.337280 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3887,"thread_start_us":3638,"threads_started":1}
I20260619 06:40:26.339890 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.299174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:36009 (local address 127.7.146.254:33983)
0619 06:40:26.315581 (+ 16407us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:26.315588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:26.315610 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:40:26.315669 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:26.315674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:26.315752 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:26.315863 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:26.317137 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.318247 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.320223 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.321151 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.321271 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.329938 (+ 8667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:26.329946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:26.329951 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:26.330012 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:26.333500 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.335229 (+ 1729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.335235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.335238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.335386 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.335793 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.335799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.335802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.336556 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:26.336697 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:26.338268 (+ 1571us) server_negotiation.cc:299] Negotiation successful
0619 06:40:26.339448 (+ 1180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16175,"thread_start_us":16872,"threads_started":1}
I20260619 06:40:26.339104 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.298963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33227 (local address 127.7.146.193:58149)
0619 06:40:26.301139 (+ 2176us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:26.301190 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:26.304724 (+ 3534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:26.312212 (+ 7488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:26.312227 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:26.313828 (+ 1601us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:26.314732 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.314758 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.316363 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.316370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.318605 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.318622 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.319998 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.320006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.320308 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.322862 (+ 2554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:26.322897 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:26.326357 (+ 3460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:26.332214 (+ 5857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.332224 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.332244 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.332930 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.333385 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.333389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.333393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.333661 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.334378 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:26.334387 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:26.334886 (+ 499us) client_negotiation.cc:769] Sending connection context
0619 06:40:26.336264 (+ 1378us) client_negotiation.cc:240] Negotiation successful
0619 06:40:26.336915 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1889,"thread_start_us":1653,"threads_started":1}
I20260619 06:40:26.341971 14864 heartbeater.cc:347] Connected to a master server at 127.7.146.252:33227
I20260619 06:40:26.342095 14865 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33983
I20260619 06:40:26.342253 14864 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:26.342257 14865 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:26.343312 14865 heartbeater.cc:511] Master 127.7.146.254:33983 requested a full tablet report, sending...
Jun 19 06:40:26 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851226, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.253@KRBTEST.COM
I20260619 06:40:26.346477 13898 ts_manager.cc:194] Re-registered known tserver with Master: 8ff7fd3845374e38a40f1eda191ec548 (127.7.146.193:41477)
I20260619 06:40:26.347095 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.305555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.7.146.194:33669)
0619 06:40:26.320358 (+ 14803us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:26.320380 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:26.320414 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:40:26.320482 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:26.320491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:26.320849 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:26.321002 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:26.323901 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.328135 (+ 4234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.330289 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.331286 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.331423 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.337636 (+ 6213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:26.337652 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:26.337669 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:26.337714 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:26.341694 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.342571 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.342580 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.342583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.342772 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.343927 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.343932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.343935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.344324 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:26.344776 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:26.346358 (+ 1582us) server_negotiation.cc:299] Negotiation successful
0619 06:40:26.346635 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12661,"thread_start_us":12305,"threads_started":1}
I20260619 06:40:26.347597 13898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:36009
I20260619 06:40:26.348587 14864 heartbeater.cc:511] Master 127.7.146.252:33227 requested a full tablet report, sending...
I20260619 06:40:26.348731 14216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:26.350499 14685 ts_manager.cc:194] Registered new tserver with Master: 8ff7fd3845374e38a40f1eda191ec548 (127.7.146.193:41477)
I20260619 06:40:26.355156 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.301921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:32991 (local address 127.7.146.253:35469)
0619 06:40:26.327570 (+ 25649us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:26.327577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:26.335048 (+ 7471us) server_negotiation.cc:407] Connection header received
0619 06:40:26.335195 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:26.335200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:26.335298 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:26.335410 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:26.336795 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.337826 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.339710 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.340403 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.340497 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.345247 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:26.345253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:26.345258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:26.345299 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:26.351767 (+ 6468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.352672 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.352678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.352682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.352849 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.353561 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.353566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.353570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.353928 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:26.354057 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:26.354557 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 06:40:26.354770 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25497,"thread_start_us":25364,"threads_started":1}
I20260619 06:40:26.355172 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.301821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35469 (local address 127.7.146.193:32991)
0619 06:40:26.334926 (+ 33105us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:26.334945 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:26.335078 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:26.335663 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:26.335668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:26.336199 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:26.336557 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.336638 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.338019 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.338024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.339554 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.339570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.340833 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.340902 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.341081 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.341582 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:26.341605 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:26.344966 (+ 3361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:26.351914 (+ 6948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.351920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.351925 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.352438 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.353233 (+ 795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.353238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.353243 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.353422 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.354167 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:26.354172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:26.354319 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 06:40:26.354460 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:40:26.354769 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32939,"thread_start_us":32580,"threads_started":1}
I20260619 06:40:26.356374 14866 heartbeater.cc:347] Connected to a master server at 127.7.146.253:35469
I20260619 06:40:26.356526 14866 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:26.356920 14866 heartbeater.cc:511] Master 127.7.146.253:35469 requested a full tablet report, sending...
I20260619 06:40:26.358618 14495 ts_manager.cc:194] Registered new tserver with Master: 8ff7fd3845374e38a40f1eda191ec548 (127.7.146.193:41477)
I20260619 06:40:26.437121 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 14093
I20260619 06:40:26.450017 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:33669
--local_ip_for_outbound_sockets=127.7.146.194
--tserver_master_addrs=127.7.146.254:33983
--webserver_port=42129
--webserver_interface=127.7.146.194
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab
--principal=oryx/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.146.254:33983,127.7.146.253:35469
--tserver_master_addrs=127.7.146.254:33983,127.7.146.253:35469,127.7.146.252:33227 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:40:26.602540 14639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: NON_VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: true } } }
I20260619 06:40:26.602969 14639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:26.604990 14877 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a: attempting to promote NON_VOTER 354e9743e7cb4872ac5c3b3fe6e5f8b6 to VOTER
I20260619 06:40:26.605535 14877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } }
I20260619 06:40:26.607318 14695 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 LEARNER]: Refusing update from remote peer 871d0d204466410e8683d77affe62f3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 06:40:26.607600 14505 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 FOLLOWER]: Refusing update from remote peer 871d0d204466410e8683d77affe62f3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 06:40:26.608088 14878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 06:40:26.608700 14877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 06:40:26.611245 14878 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 354e9743e7cb4872ac5c3b3fe6e5f8b6 (127.7.146.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:26.611918 14505 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 354e9743e7cb4872ac5c3b3fe6e5f8b6 (127.7.146.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:26.615805 14878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:26.616201 14878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:26.616526 14639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:26.616827 14639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:26.619098 14639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:26.619405 14639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:26.620674 14695 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 354e9743e7cb4872ac5c3b3fe6e5f8b6 (127.7.146.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:26.623509 14881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:26.623972 14881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:40:26.624868 14723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "871d0d204466410e8683d77affe62f3a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:26.625062 14723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 06:40:26.773952 14876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:26.774243 14876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:26.774366 14876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:26.781653 14876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:26.781780 14876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:26.781930 14876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:40:26.791646 14876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab
--principal=oryx/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:33669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=42129
--tserver_master_addrs=127.7.146.254:33983,127.7.146.253:35469,127.7.146.252:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:26.793045 14876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:26.794385 14876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:26.807885 14889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:26.807988 14886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:26.807977 14887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:26.808677 14876 server_base.cc:1034] running on GCE node
I20260619 06:40:26.809257 14876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:26.810161 14876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:26.811373 14876 hybrid_clock.cc:648] HybridClock initialized: now 1781851226811318 us; error 40 us; skew 500 ppm
I20260619 06:40:26.811733 14876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:26 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851226, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:26.815685 14876 init.cc:377] Logged in from keytab as oryx/127.7.146.194@KRBTEST.COM (short username oryx)
I20260619 06:40:26.817559 14876 webserver.cc:460] Webserver started at http://127.7.146.194:42129/ using document root <none> and password file <none>
I20260619 06:40:26.818424 14876 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:26.818574 14876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:26.824784 14876 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:40:26.827497 14896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:26.829030 14876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:26.829178 14876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "7022dfc23a67495c9eac0af30f5ce398"
format_stamp: "Formatted at 2026-06-19 06:40:23 on dist-test-slave-qfk4"
server_key: "8eaccfcd5255c6d69f290fd0b9aee613"
server_key_iv: "a190e333ce67dda0ee676d36554dabba"
server_key_version: "encryptionkey@0"
I20260619 06:40:26.830127 14876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:26.840866 14876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:26.845134 14876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:26.845381 14876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:26.846312 14876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:26.847654 14876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:26.847745 14876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:26.847844 14876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:26.847913 14876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:26.877974 14876 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:33669
I20260619 06:40:26.877993 15008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:33669 every 8 connection(s)
I20260619 06:40:26.879690 14876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:40:26.888976 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 14876
I20260619 06:40:26.889276 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 14226
Jun 19 06:40:26 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851226, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.252@KRBTEST.COM
Jun 19 06:40:26 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.146.195@KRBTEST.COM
Jun 19 06:40:26 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851226, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:26.929034 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.890317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.7.146.195:32911)
0619 06:40:26.902996 (+ 12679us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:26.903006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:26.903037 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:40:26.903107 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:26.903116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:26.903380 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:26.903531 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:26.906482 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.908210 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.911301 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.912270 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.912384 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.921620 (+ 9236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:26.921642 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:26.921687 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:26.921736 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:26.925635 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.926463 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.926473 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.926476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.926639 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.927084 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.927089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.927092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.927502 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:26.927760 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:26.928328 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 06:40:26.928609 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12497,"thread_start_us":12357,"threads_started":1}
I20260619 06:40:26.930279 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.882599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:33361 (local address 127.7.146.254:33983)
0619 06:40:26.899953 (+ 17354us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:26.899958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:26.899974 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:40:26.900023 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:26.900026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:26.900093 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:26.900185 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:26.901994 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.911093 (+ 9099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.913942 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.919050 (+ 5108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.920855 (+ 1805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.923598 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:26.923602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:26.923605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:26.923631 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:26.926777 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.927862 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.927866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.927869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.928009 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.928481 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.928485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.928488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.928834 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:26.929395 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:26.929719 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 06:40:26.929947 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17258,"thread_start_us":17175,"threads_started":1}
I20260619 06:40:26.931149 14349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:26.932096 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.883227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33227 (local address 127.7.146.194:41149)
0619 06:40:26.902360 (+ 19133us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:26.902379 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:26.902486 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:26.904471 (+ 1985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:26.904476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:26.905182 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:26.905517 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.905529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.907326 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.907332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.909979 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.909997 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.913120 (+ 3123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.913132 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.913461 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.914655 (+ 1194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:26.914706 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:26.918157 (+ 3451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:26.926722 (+ 8565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.926731 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.926750 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.927377 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.927792 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.927796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.927800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.928057 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.928692 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:26.928700 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:26.930912 (+ 2212us) client_negotiation.cc:769] Sending connection context
0619 06:40:26.931055 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 06:40:26.931317 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18984,"mutex_wait_us":872,"thread_start_us":17984,"threads_started":1}
I20260619 06:40:26.932130 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.883227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.194:33361)
0619 06:40:26.897722 (+ 14495us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:26.897777 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:26.899409 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:26.900458 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:26.900468 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:26.901201 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:26.901804 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.901821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.911208 (+ 9387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.911214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.912891 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.912906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.920344 (+ 7438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.920350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.920540 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.921098 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:26.921114 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:26.923381 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:26.927239 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.927244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.927247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.927720 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.928139 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.928143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.928146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.928334 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.928967 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:26.928972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:26.929275 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 06:40:26.929580 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 06:40:26.930136 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14221,"mutex_wait_us":23,"thread_start_us":14005,"threads_started":1}
I20260619 06:40:26.933379 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.883481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:41149 (local address 127.7.146.252:33227)
0619 06:40:26.904130 (+ 20649us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:26.904136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:26.904156 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:40:26.904215 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:26.904221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:26.904295 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:26.905673 (+ 1378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:26.905845 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.907128 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.910203 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.910925 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.919248 (+ 8323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.919572 (+ 324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:26.919578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:26.919584 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:26.919616 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:26.926584 (+ 6968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.927538 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.927542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.927546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.927677 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.928200 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.928205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.928209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.928521 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:26.930853 (+ 2332us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:26.932808 (+ 1955us) server_negotiation.cc:299] Negotiation successful
0619 06:40:26.933037 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20515,"thread_start_us":20370,"threads_started":1}
Jun 19 06:40:26 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851226, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.253@KRBTEST.COM
I20260619 06:40:26.933872 15010 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33983
I20260619 06:40:26.934132 15010 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:26.934906 15010 heartbeater.cc:511] Master 127.7.146.254:33983 requested a full tablet report, sending...
I20260619 06:40:26.935946 15009 heartbeater.cc:347] Connected to a master server at 127.7.146.252:33227
I20260619 06:40:26.936049 15009 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:26.936378 15009 heartbeater.cc:511] Master 127.7.146.252:33227 requested a full tablet report, sending...
I20260619 06:40:26.936390 13898 ts_manager.cc:194] Re-registered known tserver with Master: 7022dfc23a67495c9eac0af30f5ce398 (127.7.146.194:33669)
I20260619 06:40:26.937295 13898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:33361
I20260619 06:40:26.937466 14685 ts_manager.cc:194] Registered new tserver with Master: 7022dfc23a67495c9eac0af30f5ce398 (127.7.146.194:33669)
I20260619 06:40:26.944453 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.902692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35469 (local address 127.7.146.194:36287)
0619 06:40:26.924774 (+ 22082us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:26.924792 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:26.925197 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:26.925768 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:26.925772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:26.926264 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:26.926572 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.926581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.927950 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.927955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.929526 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:26.929539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.930588 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.930594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:26.930781 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.931373 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:26.931393 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:26.937375 (+ 5982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:26.941623 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.941628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.941632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.942088 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.942583 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:26.942588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:26.942592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:26.942754 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:26.943402 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:26.943408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:26.943599 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 06:40:26.943718 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 06:40:26.944027 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21944,"mutex_wait_us":51,"thread_start_us":21105,"threads_started":1}
I20260619 06:40:26.945466 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:26.902719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:36287 (local address 127.7.146.253:35469)
0619 06:40:26.920143 (+ 17424us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:26.920151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:26.924878 (+ 4727us) server_negotiation.cc:407] Connection header received
0619 06:40:26.925361 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:26.925366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:26.925480 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:26.925561 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:26.926728 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.927789 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.929674 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:26.930238 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:26.930325 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:26.937619 (+ 7294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:26.937627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:26.937632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:26.937698 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:26.941495 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.942237 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.942243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.942247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.942395 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:26.942886 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:26.942891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:26.942894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:26.943274 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:26.944743 (+ 1469us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:26.944892 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 06:40:26.945111 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17275,"thread_start_us":17139,"threads_started":1}
I20260619 06:40:26.946259 15011 heartbeater.cc:347] Connected to a master server at 127.7.146.253:35469
I20260619 06:40:26.946390 15011 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:26.946738 15011 heartbeater.cc:511] Master 127.7.146.253:35469 requested a full tablet report, sending...
I20260619 06:40:26.947724 14495 ts_manager.cc:194] Registered new tserver with Master: 7022dfc23a67495c9eac0af30f5ce398 (127.7.146.194:33669)
I20260619 06:40:27.023203 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 14226
I20260619 06:40:27.035777 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:32911
--local_ip_for_outbound_sockets=127.7.146.195
--tserver_master_addrs=127.7.146.254:33983
--webserver_port=36123
--webserver_interface=127.7.146.195
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab
--principal=oryx/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.146.254:33983,127.7.146.253:35469
--tserver_master_addrs=127.7.146.254:33983,127.7.146.253:35469,127.7.146.252:33227 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:40:27.351207 14865 heartbeater.cc:503] Master 127.7.146.254:33983 was elected leader, sending a full tablet report...
W20260619 06:40:27.358268 15022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:27.358536 15022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:27.358652 15022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:27.366091 15022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:27.366197 15022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:27.366335 15022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:40:27.376333 15022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab
--principal=oryx/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:32911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=36123
--tserver_master_addrs=127.7.146.254:33983,127.7.146.253:35469,127.7.146.252:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:27.377683 15022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:27.379010 15022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:27.392267 15030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:27.392319 15028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:27.392302 15027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:27.392704 15022 server_base.cc:1034] running on GCE node
I20260619 06:40:27.393560 15022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:27.394469 15022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:27.395716 15022 hybrid_clock.cc:648] HybridClock initialized: now 1781851227395656 us; error 45 us; skew 500 ppm
I20260619 06:40:27.396062 15022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:27 dist-test-slave-qfk4 krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851227, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:27.399194 15022 init.cc:377] Logged in from keytab as oryx/127.7.146.195@KRBTEST.COM (short username oryx)
I20260619 06:40:27.400720 15022 webserver.cc:460] Webserver started at http://127.7.146.195:36123/ using document root <none> and password file <none>
I20260619 06:40:27.401407 15022 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:27.401542 15022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:27.406994 15022 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:40:27.409766 15037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:27.411123 15022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:40:27.411263 15022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "7bf794b451df4f7a999848e2c06d0457"
format_stamp: "Formatted at 2026-06-19 06:40:24 on dist-test-slave-qfk4"
server_key: "54d8d27cf06b4409b5304068f9590630"
server_key_iv: "6411d26b101d649a0192cddd701abf97"
server_key_version: "encryptionkey@0"
I20260619 06:40:27.412221 15022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:27.429023 15022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:27.433104 15022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:27.433349 15022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:27.434227 15022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:27.435570 15022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:27.435657 15022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:27.435738 15022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:27.435789 15022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:27.465776 15022 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:32911
I20260619 06:40:27.465829 15149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:32911 every 8 connection(s)
I20260619 06:40:27.467237 15022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:40:27.475555 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 15022
Jun 19 06:40:27 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851227, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.252@KRBTEST.COM
Jun 19 06:40:27 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851227, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:27.522645 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.470513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:57287 (local address 127.7.146.254:33983)
0619 06:40:27.489316 (+ 18803us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.489323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.490156 (+ 833us) server_negotiation.cc:407] Connection header received
0619 06:40:27.493857 (+ 3701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.493864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.493980 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.494065 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:27.496990 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.498269 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.501402 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.502028 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.502118 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.512909 (+ 10791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:27.512917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:27.512923 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:27.512983 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:27.517125 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.519199 (+ 2074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.519205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.519210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.519383 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.519878 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.519883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.519886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.520256 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:27.520400 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.521903 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.522199 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18657,"thread_start_us":18550,"threads_started":1}
I20260619 06:40:27.523538 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.470584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.195:57287)
0619 06:40:27.489969 (+ 19385us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:27.490023 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:27.493464 (+ 3441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:27.494579 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:27.494594 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:27.496403 (+ 1809us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:27.496760 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.496786 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.498479 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.498485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.501249 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.501261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.502470 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.502478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.502677 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.507714 (+ 5037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:27.507743 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:27.511760 (+ 4017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:27.517307 (+ 5547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:27.517318 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:27.517340 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:27.518076 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:27.519485 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:27.519490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:27.519495 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:27.519781 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:27.520365 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:27.520374 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:27.520921 (+ 547us) client_negotiation.cc:769] Sending connection context
0619 06:40:27.521288 (+ 367us) client_negotiation.cc:240] Negotiation successful
0619 06:40:27.521927 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19056,"mutex_wait_us":910,"thread_start_us":19596,"threads_started":1}
I20260619 06:40:27.525561 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.470584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33227 (local address 127.7.146.195:37879)
0619 06:40:27.490850 (+ 20266us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:27.490872 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:27.493470 (+ 2598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:27.494579 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:27.494594 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:27.495743 (+ 1149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:27.496760 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.496787 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.498469 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.498474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.500962 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.500987 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.501956 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.501976 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.502290 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.503107 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:27.503141 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:27.508010 (+ 4869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:27.518119 (+ 10109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:27.518125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:27.518129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:27.518675 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:27.520842 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:27.520848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:27.520853 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:27.521052 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:27.524383 (+ 3331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:27.524390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:27.524597 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 06:40:27.524765 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 06:40:27.525089 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20102,"mutex_wait_us":102,"spinlock_wait_cycles":43392,"thread_start_us":18765,"threads_started":1}
I20260619 06:40:27.526708 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.469889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:37879 (local address 127.7.146.252:33227)
0619 06:40:27.485337 (+ 15448us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.485345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.490997 (+ 5652us) server_negotiation.cc:407] Connection header received
0619 06:40:27.493989 (+ 2992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.493996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.494087 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.494188 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:27.497004 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.498241 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.501133 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.501736 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.501853 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.510383 (+ 8530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:27.510393 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:27.510400 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:27.510453 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:27.517874 (+ 7421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.518852 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.518859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.518863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.519034 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.521201 (+ 2167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.521208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.521212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.521582 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:27.521728 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.526056 (+ 4328us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.526317 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15276,"thread_start_us":15463,"threads_started":1}
I20260619 06:40:27.529834 15150 heartbeater.cc:347] Connected to a master server at 127.7.146.252:33227
I20260619 06:40:27.530194 15150 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:27.531160 15150 heartbeater.cc:511] Master 127.7.146.252:33227 requested a full tablet report, sending...
I20260619 06:40:27.533255 14685 ts_manager.cc:194] Registered new tserver with Master: 7bf794b451df4f7a999848e2c06d0457 (127.7.146.195:32911)
I20260619 06:40:27.537606 15151 heartbeater.cc:347] Connected to a master server at 127.7.146.254:33983
I20260619 06:40:27.537787 15151 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:27.538281 15151 heartbeater.cc:511] Master 127.7.146.254:33983 requested a full tablet report, sending...
I20260619 06:40:27.539331 13898 ts_manager.cc:194] Re-registered known tserver with Master: 7bf794b451df4f7a999848e2c06d0457 (127.7.146.195:32911)
I20260619 06:40:27.540481 13898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:57287
Jun 19 06:40:27 dist-test-slave-qfk4 krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851227, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.253@KRBTEST.COM
I20260619 06:40:27.548985 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.490922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:52161 (local address 127.7.146.253:35469)
0619 06:40:27.499486 (+ 8564us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.499491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.499506 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:40:27.499562 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.499565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.499627 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.499722 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:27.528666 (+ 28944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.534589 (+ 5923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.536928 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.537554 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.537747 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.542180 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:27.542188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:27.542193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:27.542248 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:27.546277 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.546886 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.546891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.546895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.547044 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.547466 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.547472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.547476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.547836 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:27.547990 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.548377 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.548606 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8461,"thread_start_us":8367,"threads_started":1}
I20260619 06:40:27.549016 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.491597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35469 (local address 127.7.146.195:52161)
0619 06:40:27.495215 (+ 3618us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:27.495234 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:27.495355 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:27.499785 (+ 4430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:27.499789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:27.500396 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:27.500785 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.500797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.534858 (+ 34061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.534865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.536686 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.536709 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.538261 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.538272 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.538474 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.539168 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:27.539192 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:27.541989 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:27.546400 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:27.546405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:27.546410 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:27.546757 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:27.547153 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:27.547158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:27.547162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:27.547325 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:27.547954 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:27.547958 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:27.548139 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 06:40:27.548253 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 06:40:27.548598 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3480,"thread_start_us":4148,"threads_started":1}
I20260619 06:40:27.550208 15152 heartbeater.cc:347] Connected to a master server at 127.7.146.253:35469
I20260619 06:40:27.550360 15152 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:27.550819 15152 heartbeater.cc:511] Master 127.7.146.253:35469 requested a full tablet report, sending...
I20260619 06:40:27.552019 14495 ts_manager.cc:194] Registered new tserver with Master: 7bf794b451df4f7a999848e2c06d0457 (127.7.146.195:32911)
I20260619 06:40:27.555645 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.501537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.7.146.254:33983)
0619 06:40:27.523285 (+ 21748us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.523291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.537743 (+ 14452us) server_negotiation.cc:407] Connection header received
0619 06:40:27.538512 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.538518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.538608 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.538696 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:27.541279 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.542299 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.544196 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.544798 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.544912 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.548167 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:27.548172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:27.548175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:27.548206 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:27.551475 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.553462 (+ 1987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.553467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.553471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.553610 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.554090 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.554094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.554096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.554353 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:27.554606 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.555039 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.555265 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21631,"thread_start_us":35208,"threads_started":1}
I20260619 06:40:27.559341 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.502190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.7.146.253:35469)
0619 06:40:27.537186 (+ 34996us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.537192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.538545 (+ 1353us) server_negotiation.cc:407] Connection header received
0619 06:40:27.538600 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.538604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.538711 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.538830 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:27.541455 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.545228 (+ 3773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.547177 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.547858 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.547977 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.552731 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:27.552737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:27.552742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:27.552778 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:27.556054 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.557170 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.557175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.557178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.557333 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.557803 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.557808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.557810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.558141 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:27.558361 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.558689 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.559078 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34845,"thread_start_us":34704,"threads_started":1}
I20260619 06:40:27.562095 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.538059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.7.146.252:33227)
0619 06:40:27.538220 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.538225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.544767 (+ 6542us) server_negotiation.cc:407] Connection header received
0619 06:40:27.545513 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.545519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.545606 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.545701 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:27.550066 (+ 4365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.550838 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.552542 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.552989 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.553099 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.556445 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:27.556452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:27.556455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:27.556494 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:27.559586 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.560207 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.560213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.560216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.560359 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.560767 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.560772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.560775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.561086 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:27.561312 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.561571 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.561784 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:40:27.559675 13898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:40:27.567530 13898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:40:27.599388 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.589951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.7.146.194:33669)
0619 06:40:27.590642 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.590651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.591241 (+ 590us) server_negotiation.cc:407] Connection header received
0619 06:40:27.591422 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.591431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.591727 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.591923 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:27.593276 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.594349 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.596714 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.598037 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.598152 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.598354 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.598922 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.599108 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":631,"threads_started":1}
I20260619 06:40:27.599395 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.589719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:33669 (local address 127.0.0.1:37824)
0619 06:40:27.591096 (+ 1377us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:27.591118 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:27.591269 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:27.592147 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:27.592151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:27.592176 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:27.593070 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.593083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.594480 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.594485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.596540 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.596556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.598497 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.598504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.598723 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.598756 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:40:27.598857 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:40:27.598991 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"mutex_wait_us":19,"thread_start_us":1075,"threads_started":1}
I20260619 06:40:27.600711 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.590109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:41477 (local address 127.0.0.1:55020)
0619 06:40:27.593490 (+ 3381us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:27.593504 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:27.593599 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:27.594509 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:27.594513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:27.594547 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:27.595008 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.595020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.596757 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.596761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.598301 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.598313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.599965 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.599972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.600124 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.600155 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:40:27.600234 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:40:27.600353 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1546,"thread_start_us":1404,"threads_started":1}
I20260619 06:40:27.600739 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.590460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.7.146.193:41477)
0619 06:40:27.594038 (+ 3578us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.594045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.594068 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:40:27.594143 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.594149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.594333 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.594503 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:27.595183 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.596652 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.598459 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.599768 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.599875 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.600098 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.600284 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.600438 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3424,"thread_start_us":3326,"threads_started":1}
I20260619 06:40:27.603710 14944 tablet_service.cc:1467] Processing CreateTablet for tablet f3891ae5bb004857a48bfbb21eda88ef (DEFAULT_TABLE table=test-table [id=c003168c20074056b513693a46e29d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:27.604517 14799 tablet_service.cc:1467] Processing CreateTablet for tablet f3891ae5bb004857a48bfbb21eda88ef (DEFAULT_TABLE table=test-table [id=c003168c20074056b513693a46e29d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:27.605280 14944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3891ae5bb004857a48bfbb21eda88ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:27.605883 14799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3891ae5bb004857a48bfbb21eda88ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:27.610307 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.591857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:32911 (local address 127.0.0.1:58792)
0619 06:40:27.593228 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:27.593248 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:27.593370 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:27.595427 (+ 2057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:27.595431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:27.595463 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:27.595814 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.595822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.597978 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.597983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.599798 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.599812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.609406 (+ 9594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.609413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.609648 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.609676 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:40:27.609764 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:40:27.609909 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"mutex_wait_us":2,"thread_start_us":1039,"threads_started":1}
I20260619 06:40:27.610714 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.592204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.7.146.195:32911)
0619 06:40:27.594877 (+ 2673us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.594885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.594917 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:40:27.595020 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.595027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.595248 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.595412 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:27.595991 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.597612 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.599991 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.609183 (+ 9192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.609320 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.609532 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.609826 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.610132 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2512,"thread_start_us":299,"threads_started":1}
I20260619 06:40:27.613931 15085 tablet_service.cc:1467] Processing CreateTablet for tablet f3891ae5bb004857a48bfbb21eda88ef (DEFAULT_TABLE table=test-table [id=c003168c20074056b513693a46e29d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:27.615049 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3891ae5bb004857a48bfbb21eda88ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:27.625242 15184 tablet_bootstrap.cc:492] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548: Bootstrap starting.
I20260619 06:40:27.625648 15183 tablet_bootstrap.cc:492] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398: Bootstrap starting.
I20260619 06:40:27.626883 15185 tablet_bootstrap.cc:492] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: Bootstrap starting.
I20260619 06:40:27.629092 15184 tablet_bootstrap.cc:654] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:27.630079 15185 tablet_bootstrap.cc:654] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:27.630110 15184 log.cc:824] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:27.631248 15185 log.cc:824] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:27.631773 15183 tablet_bootstrap.cc:654] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:27.632871 15184 tablet_bootstrap.cc:492] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548: No bootstrap required, opened a new log
I20260619 06:40:27.632833 15183 log.cc:824] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:27.633116 15184 ts_tablet_manager.cc:1397] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 06:40:27.634492 15185 tablet_bootstrap.cc:492] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: No bootstrap required, opened a new log
I20260619 06:40:27.634784 15185 ts_tablet_manager.cc:1397] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 06:40:27.635915 15183 tablet_bootstrap.cc:492] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398: No bootstrap required, opened a new log
I20260619 06:40:27.636245 15183 ts_tablet_manager.cc:1397] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260619 06:40:27.641287 15184 raft_consensus.cc:348] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } }
I20260619 06:40:27.641641 15184 raft_consensus.cc:374] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:27.641728 15184 raft_consensus.cc:729] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff7fd3845374e38a40f1eda191ec548, State: Initialized, Role: FOLLOWER
I20260619 06:40:27.642299 15184 consensus_queue.cc:260] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } }
I20260619 06:40:27.645153 15185 raft_consensus.cc:348] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } }
I20260619 06:40:27.646024 15184 ts_tablet_manager.cc:1428] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260619 06:40:27.645623 15185 raft_consensus.cc:374] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:27.646596 15185 raft_consensus.cc:729] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf794b451df4f7a999848e2c06d0457, State: Initialized, Role: FOLLOWER
I20260619 06:40:27.647298 15185 consensus_queue.cc:260] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } }
I20260619 06:40:27.647601 15183 raft_consensus.cc:348] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } }
I20260619 06:40:27.648016 15183 raft_consensus.cc:374] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:27.648103 15183 raft_consensus.cc:729] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7022dfc23a67495c9eac0af30f5ce398, State: Initialized, Role: FOLLOWER
I20260619 06:40:27.648810 15183 consensus_queue.cc:260] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } }
I20260619 06:40:27.651921 15183 ts_tablet_manager.cc:1428] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260619 06:40:27.652206 15010 heartbeater.cc:503] Master 127.7.146.254:33983 was elected leader, sending a full tablet report...
I20260619 06:40:27.655711 15151 heartbeater.cc:503] Master 127.7.146.254:33983 was elected leader, sending a full tablet report...
I20260619 06:40:27.655963 15185 ts_tablet_manager.cc:1428] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: Time spent starting tablet: real 0.021s user 0.015s sys 0.005s
I20260619 06:40:27.663875 15191 raft_consensus.cc:482] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:40:27.664121 15191 raft_consensus.cc:504] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } }
I20260619 06:40:27.665786 15191 leader_election.cc:288] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ff7fd3845374e38a40f1eda191ec548 (127.7.146.193:41477), 7bf794b451df4f7a999848e2c06d0457 (127.7.146.195:32911)
I20260619 06:40:27.672761 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.666183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:51155 (local address 127.7.146.193:41477)
0619 06:40:27.666341 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.666345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.666670 (+ 325us) server_negotiation.cc:407] Connection header received
0619 06:40:27.666902 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.666908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.666982 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.667073 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:27.667677 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.668639 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.670909 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.671834 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.672052 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.672126 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.672361 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.672491 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:40:27.672773 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.666021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:41477 (local address 127.7.146.194:51155)
0619 06:40:27.666582 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:27.666601 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:27.666731 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:27.667110 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:27.667116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:27.667153 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:27.667552 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.667562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.668771 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.668776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.670740 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.670754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.672012 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.672017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.672211 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.672245 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:40:27.672321 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:40:27.672413 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":23,"thread_start_us":272,"threads_started":1}
I20260619 06:40:27.673552 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.666785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:47405 (local address 127.7.146.195:32911)
0619 06:40:27.666955 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.666960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.667699 (+ 739us) server_negotiation.cc:407] Connection header received
0619 06:40:27.667856 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.667861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.667930 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.668007 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:27.668454 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.669389 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.671535 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.672671 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.672819 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.672887 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.673216 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.673341 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:40:27.673601 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.666716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:32911 (local address 127.7.146.194:47405)
0619 06:40:27.667592 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:27.667609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:27.667731 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:27.668047 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:27.668051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:27.668075 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:27.668313 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.668321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.669522 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.669529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.671311 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:27.671322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.672870 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.672876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:27.673034 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.673067 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:40:27.673189 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 06:40:27.673328 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":1,"thread_start_us":585,"threads_started":1}
I20260619 06:40:27.673815 14819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3891ae5bb004857a48bfbb21eda88ef" candidate_uuid: "7022dfc23a67495c9eac0af30f5ce398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff7fd3845374e38a40f1eda191ec548" is_pre_election: true
I20260619 06:40:27.674293 14819 raft_consensus.cc:2454] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7022dfc23a67495c9eac0af30f5ce398 in term 0.
I20260619 06:40:27.674451 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3891ae5bb004857a48bfbb21eda88ef" candidate_uuid: "7022dfc23a67495c9eac0af30f5ce398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf794b451df4f7a999848e2c06d0457" is_pre_election: true
I20260619 06:40:27.674912 15105 raft_consensus.cc:2454] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7022dfc23a67495c9eac0af30f5ce398 in term 0.
I20260619 06:40:27.674917 14897 leader_election.cc:302] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7022dfc23a67495c9eac0af30f5ce398, 8ff7fd3845374e38a40f1eda191ec548; no voters:
I20260619 06:40:27.675304 15191 raft_consensus.cc:2781] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:40:27.675443 15191 raft_consensus.cc:482] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:40:27.675549 15191 raft_consensus.cc:3037] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:27.677794 15191 raft_consensus.cc:504] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } }
I20260619 06:40:27.678309 15191 leader_election.cc:288] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [CANDIDATE]: Term 1 election: Requested vote from peers 8ff7fd3845374e38a40f1eda191ec548 (127.7.146.193:41477), 7bf794b451df4f7a999848e2c06d0457 (127.7.146.195:32911)
I20260619 06:40:27.678750 14819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3891ae5bb004857a48bfbb21eda88ef" candidate_uuid: "7022dfc23a67495c9eac0af30f5ce398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff7fd3845374e38a40f1eda191ec548"
I20260619 06:40:27.678853 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3891ae5bb004857a48bfbb21eda88ef" candidate_uuid: "7022dfc23a67495c9eac0af30f5ce398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf794b451df4f7a999848e2c06d0457"
I20260619 06:40:27.678925 14819 raft_consensus.cc:3037] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:27.679044 15105 raft_consensus.cc:3037] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:27.681114 14819 raft_consensus.cc:2454] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7022dfc23a67495c9eac0af30f5ce398 in term 1.
I20260619 06:40:27.681228 15105 raft_consensus.cc:2454] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7022dfc23a67495c9eac0af30f5ce398 in term 1.
I20260619 06:40:27.681468 14897 leader_election.cc:302] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7022dfc23a67495c9eac0af30f5ce398, 8ff7fd3845374e38a40f1eda191ec548; no voters:
I20260619 06:40:27.681730 15191 raft_consensus.cc:2781] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:27.682156 15191 raft_consensus.cc:686] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 1 LEADER]: Becoming Leader. State: Replica: 7022dfc23a67495c9eac0af30f5ce398, State: Running, Role: LEADER
I20260619 06:40:27.682667 15191 consensus_queue.cc:237] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } }
I20260619 06:40:27.687799 13898 catalog_manager.cc:5495] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7022dfc23a67495c9eac0af30f5ce398 (127.7.146.194). New cstate: current_term: 1 leader_uuid: "7022dfc23a67495c9eac0af30f5ce398" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } health_report { overall_health: UNKNOWN } } }
W20260619 06:40:27.720674 15153 tablet.cc:2369] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:40:27.724454 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.719365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.7.146.194:33669)
0619 06:40:27.719562 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.719568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.719611 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:40:27.719714 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.719718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.719796 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.719925 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:40:27.720297 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.721036 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.722363 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.722775 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.722882 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.723354 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:40:27.723879 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:40:27.724026 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.724118 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.724199 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:40:27.732669 15105 raft_consensus.cc:1261] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457 [term 1 FOLLOWER]: Refusing update from remote peer 7022dfc23a67495c9eac0af30f5ce398: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:27.732762 14819 raft_consensus.cc:1261] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548 [term 1 FOLLOWER]: Refusing update from remote peer 7022dfc23a67495c9eac0af30f5ce398: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:27.733698 15194 consensus_queue.cc:1035] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:27.734081 15191 consensus_queue.cc:1035] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:27.744731 15197 mvcc.cc:205] Tried to move back new op lower bound from 7298462628782493696 to 7298462628591853568. Current Snapshot: MvccSnapshot[applied={T|T < 7298462628782493696}]
I20260619 06:40:27.774730 13898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:27.775010 13898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:27.781508 13898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 871d0d204466410e8683d77affe62f3a: Sending DeleteTablet for 3 replicas of tablet f3891ae5bb004857a48bfbb21eda88ef
I20260619 06:40:27.783209 14944 tablet_service.cc:1514] Processing DeleteTablet for tablet f3891ae5bb004857a48bfbb21eda88ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:27 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:37824
I20260619 06:40:27.783262 14799 tablet_service.cc:1514] Processing DeleteTablet for tablet f3891ae5bb004857a48bfbb21eda88ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:27 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:55020
I20260619 06:40:27.784129 15206 tablet_replica.cc:330] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398: stopping tablet replica
I20260619 06:40:27.784168 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet f3891ae5bb004857a48bfbb21eda88ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:27 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:58792
I20260619 06:40:27.784682 15206 raft_consensus.cc:2229] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:40:27.785415 15206 raft_consensus.cc:2258] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:27.787844 15206 ts_tablet_manager.cc:1905] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:27.797191 15206 ts_tablet_manager.cc:1918] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:27.797443 15206 log.cc:1196] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/f3891ae5bb004857a48bfbb21eda88ef
I20260619 06:40:27.798017 15206 ts_tablet_manager.cc:1939] T f3891ae5bb004857a48bfbb21eda88ef P 7022dfc23a67495c9eac0af30f5ce398: Deleting consensus metadata
I20260619 06:40:27.798032 15208 tablet_replica.cc:330] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: stopping tablet replica
I20260619 06:40:27.798339 15207 tablet_replica.cc:330] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548: stopping tablet replica
I20260619 06:40:27.798542 15208 raft_consensus.cc:2229] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:27.798816 15207 raft_consensus.cc:2229] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:27.799086 15208 raft_consensus.cc:2258] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:27.799345 15207 raft_consensus.cc:2258] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:27.800000 13884 catalog_manager.cc:4841] TS 7022dfc23a67495c9eac0af30f5ce398 (127.7.146.194:33669): tablet f3891ae5bb004857a48bfbb21eda88ef (table test-table [id=c003168c20074056b513693a46e29d1c]) successfully deleted
I20260619 06:40:27.801456 15208 ts_tablet_manager.cc:1905] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:27.802217 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 13866
I20260619 06:40:27.803100 15207 ts_tablet_manager.cc:1905] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:27.803233 13928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:27.808802 15208 ts_tablet_manager.cc:1918] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:27.808982 15208 log.cc:1196] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/f3891ae5bb004857a48bfbb21eda88ef
I20260619 06:40:27.809544 15208 ts_tablet_manager.cc:1939] T f3891ae5bb004857a48bfbb21eda88ef P 7bf794b451df4f7a999848e2c06d0457: Deleting consensus metadata
I20260619 06:40:27.810598 15207 ts_tablet_manager.cc:1918] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:27.810752 15207 log.cc:1196] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/f3891ae5bb004857a48bfbb21eda88ef
I20260619 06:40:27.811269 15207 ts_tablet_manager.cc:1939] T f3891ae5bb004857a48bfbb21eda88ef P 8ff7fd3845374e38a40f1eda191ec548: Deleting consensus metadata
I20260619 06:40:27.829497 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.787364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.7.146.253:35469)
0619 06:40:27.787565 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.787572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.800731 (+ 13159us) server_negotiation.cc:407] Connection header received
0619 06:40:27.800950 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.800955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.801042 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.801145 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:27.811989 (+ 10844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.813133 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.815730 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.816451 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.816620 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.823120 (+ 6500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:27.823128 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:27.823133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:27.823185 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:27.826575 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.827451 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.827457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.827460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.827601 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.827993 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.827997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.828000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.828341 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:27.828618 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.828937 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.829173 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:40:27.846865 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.801418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.7.146.252:33227)
0619 06:40:27.801590 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.801596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.829672 (+ 28076us) server_negotiation.cc:407] Connection header received
0619 06:40:27.829949 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.829956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.830040 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.830110 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:27.832477 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.833704 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.835519 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.836189 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.836340 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.839953 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:27.839960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:27.839966 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:27.840009 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:27.843532 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.844675 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.844682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.844686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.844845 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.845276 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.845281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.845284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.845644 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:27.845885 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.846226 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.846459 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:40:27.943006 13882 catalog_manager.cc:4841] TS 7bf794b451df4f7a999848e2c06d0457 (127.7.146.195:32911): tablet f3891ae5bb004857a48bfbb21eda88ef (table test-table [id=c003168c20074056b513693a46e29d1c]) successfully deleted
W20260619 06:40:27.943279 13882 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.146.195:32911, user_credentials={real_user=slave}} blocked reactor thread for 131866us
I20260619 06:40:27.943768 13882 catalog_manager.cc:4841] TS 8ff7fd3845374e38a40f1eda191ec548 (127.7.146.193:41477): tablet f3891ae5bb004857a48bfbb21eda88ef (table test-table [id=c003168c20074056b513693a46e29d1c]) successfully deleted
I20260619 06:40:27.944878 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 13866
I20260619 06:40:27.988807 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.971102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.7.146.253:35469)
0619 06:40:27.971294 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.971301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.972556 (+ 1255us) server_negotiation.cc:407] Connection header received
0619 06:40:27.972923 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.972931 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.973014 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.973083 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:27.975354 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.976469 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.978032 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.978559 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.978701 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.981893 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:27.981901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:27.981906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:27.981952 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:27.985414 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.986202 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.986207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.986211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.986360 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.986823 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.986828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.986831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.987179 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:27.987624 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.988210 (+ 586us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.988434 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:40:27.991693 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:27.972645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.7.146.252:33227)
0619 06:40:27.972841 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:27.972848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:27.975788 (+ 2940us) server_negotiation.cc:407] Connection header received
0619 06:40:27.976031 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:27.976039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:27.976125 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:27.976247 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:27.978415 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.979324 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.982048 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:27.982684 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:27.982810 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:27.985884 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:27.985889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:27.985892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:27.985926 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:27.988858 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.989661 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.989668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.989672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.989812 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:27.990255 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:27.990260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:27.990264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:27.990606 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:27.990842 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:27.991122 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 06:40:27.991328 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:40:28.752749 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:28.751889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.195:40805)
0619 06:40:28.752392 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:28.752520 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":279,"threads_started":1}
W20260619 06:40:28.753214 15151 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:33983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
I20260619 06:40:28.759541 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:28.758669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.194:50283)
0619 06:40:28.759219 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:28.759359 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":317,"threads_started":1}
W20260619 06:40:28.759891 15010 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:33983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
I20260619 06:40:28.774941 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:28.774158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.193:60667)
0619 06:40:28.774645 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:28.774779 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":244,"threads_started":1}
W20260619 06:40:28.775322 14865 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:33983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
I20260619 06:40:29.282496 15220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 871d0d204466410e8683d77affe62f3a)
I20260619 06:40:29.282775 15220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } }
I20260619 06:40:29.283061 15221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 871d0d204466410e8683d77affe62f3a)
I20260619 06:40:29.283360 15221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } }
I20260619 06:40:29.284262 15220 leader_election.cc:288] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 871d0d204466410e8683d77affe62f3a (127.7.146.254:33983), 354e9743e7cb4872ac5c3b3fe6e5f8b6 (127.7.146.252:33227)
I20260619 06:40:29.284741 15221 leader_election.cc:288] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 871d0d204466410e8683d77affe62f3a (127.7.146.254:33983), 61e84be559624a4586895c0acee1544c (127.7.146.253:35469)
I20260619 06:40:29.286417 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.284805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:34036)
0619 06:40:29.285856 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.286119 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":707,"threads_started":1}
W20260619 06:40:29.286783 14670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
I20260619 06:40:29.294880 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.284520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33227 (local address 127.0.0.1:59834)
0619 06:40:29.285160 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.285188 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:29.285334 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:29.286628 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:29.286636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:29.286673 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:29.287107 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:29.287122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.289263 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.289271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:29.291634 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:29.291656 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.293781 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.293790 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:29.294018 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.294070 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 06:40:29.294154 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 06:40:29.294322 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":340,"threads_started":1}
I20260619 06:40:29.295529 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.284522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:34030)
0619 06:40:29.295203 (+ 10681us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.295362 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10554,"mutex_wait_us":151,"threads_started":1}
I20260619 06:40:29.296696 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.284995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.7.146.252:33227)
0619 06:40:29.286218 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:29.286225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:29.286245 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:40:29.286321 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:29.286328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:29.286426 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:29.287300 (+ 874us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:29.287535 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.289058 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.291953 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.293535 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.295941 (+ 2406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.296082 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:29.296151 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 06:40:29.296310 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":875,"threads_started":1}
W20260619 06:40:29.298396 14480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
I20260619 06:40:29.299286 14695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61e84be559624a4586895c0acee1544c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" is_pre_election: true
I20260619 06:40:29.299649 14695 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e84be559624a4586895c0acee1544c in term 1.
I20260619 06:40:29.299965 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.297908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:34052)
0619 06:40:29.298873 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.299609 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":827}
W20260619 06:40:29.300185 14670 leader_election.cc:334] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 871d0d204466410e8683d77affe62f3a (127.7.146.254:33983): Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
I20260619 06:40:29.300254 14481 leader_election.cc:302] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354e9743e7cb4872ac5c3b3fe6e5f8b6, 61e84be559624a4586895c0acee1544c; no voters:
I20260619 06:40:29.300664 15220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 06:40:29.300796 15220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 871d0d204466410e8683d77affe62f3a)
I20260619 06:40:29.300689 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.299992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:34062)
0619 06:40:29.300181 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.300390 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:40:29.300890 15220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 1 FOLLOWER]: Advancing to term 2
W20260619 06:40:29.301028 14480 leader_election.cc:334] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 871d0d204466410e8683d77affe62f3a (127.7.146.254:33983): Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
I20260619 06:40:29.303205 15220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } }
I20260619 06:40:29.303853 15220 leader_election.cc:288] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [CANDIDATE]: Term 2 election: Requested vote from peers 871d0d204466410e8683d77affe62f3a (127.7.146.254:33983), 354e9743e7cb4872ac5c3b3fe6e5f8b6 (127.7.146.252:33227)
I20260619 06:40:29.304302 14695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61e84be559624a4586895c0acee1544c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6"
I20260619 06:40:29.304620 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.304251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:34064)
0619 06:40:29.304372 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.304459 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 06:40:29.304553 14695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:40:29.304756 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.297508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.7.146.253:35469)
0619 06:40:29.298520 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:29.298528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:29.298553 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:40:29.298627 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:29.298632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:29.298710 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:29.298820 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:29.299699 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.301072 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.302688 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.303607 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.303793 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.303943 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:29.304214 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 06:40:29.304346 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":611,"threads_started":1}
I20260619 06:40:29.305029 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.286178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35469 (local address 127.0.0.1:57296)
0619 06:40:29.292835 (+ 6657us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.292859 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:29.293031 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:29.299108 (+ 6077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:29.299115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:29.299149 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:29.299461 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:29.299475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.301254 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.301258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:29.302531 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:29.302547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.303819 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.303825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:29.304047 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.304074 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:40:29.304174 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:40:29.304335 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6542,"thread_start_us":12420,"threads_started":1}
I20260619 06:40:29.305684 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.305330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:34080)
0619 06:40:29.305450 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.305524 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 06:40:29.305917 14480 leader_election.cc:334] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 871d0d204466410e8683d77affe62f3a (127.7.146.254:33983): Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
I20260619 06:40:29.305871 14505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "61e84be559624a4586895c0acee1544c" is_pre_election: true
I20260619 06:40:29.306244 14505 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 354e9743e7cb4872ac5c3b3fe6e5f8b6 in current term 2: Already voted for candidate 61e84be559624a4586895c0acee1544c in this term.
I20260619 06:40:29.306908 14671 leader_election.cc:302] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 354e9743e7cb4872ac5c3b3fe6e5f8b6; no voters: 61e84be559624a4586895c0acee1544c, 871d0d204466410e8683d77affe62f3a
I20260619 06:40:29.307548 14695 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e84be559624a4586895c0acee1544c in term 2.
I20260619 06:40:29.307732 15221 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:40:29.307947 14481 leader_election.cc:302] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 354e9743e7cb4872ac5c3b3fe6e5f8b6, 61e84be559624a4586895c0acee1544c; no voters: 871d0d204466410e8683d77affe62f3a
I20260619 06:40:29.308244 15220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:40:29.308683 15220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [term 2 LEADER]: Becoming Leader. State: Replica: 61e84be559624a4586895c0acee1544c, State: Running, Role: LEADER
I20260619 06:40:29.309084 15220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } }
I20260619 06:40:29.311316 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61e84be559624a4586895c0acee1544c. Latest consensus state: current_term: 2 leader_uuid: "61e84be559624a4586895c0acee1544c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:29.311614 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:29.312232 15232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:29.315833 15232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:29.316671 15232 catalog_manager.cc:1259] Loaded cluster ID: eb2f1e620f71414cb0726f2011d30b04
I20260619 06:40:29.316740 15232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:29.317905 15232 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:29.318861 15232 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: Loaded TSK: 0
I20260619 06:40:29.319921 15232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:40:29.378758 14494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:40:29.385627 14494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:40:29.389629 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.389246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:34102)
0619 06:40:29.389391 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.389472 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:40:29.389853 14695 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [term 2 FOLLOWER]: Refusing update from remote peer 61e84be559624a4586895c0acee1544c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 06:40:29.390511 15230 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [LEADER]: Connected to new peer: Peer: permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 06:40:29.390695 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.390264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.0.0.1:34116)
0619 06:40:29.390422 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.390511 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:40:29.390925 14480 consensus_peers.cc:487] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c -> Peer 871d0d204466410e8683d77affe62f3a (127.7.146.254:33983): Couldn't send request to peer 871d0d204466410e8683d77affe62f3a. Status: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:40:29.393748 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61e84be559624a4586895c0acee1544c. Latest consensus state: current_term: 2 leader_uuid: "61e84be559624a4586895c0acee1544c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:29.393965 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:40:29.394758 15220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "61e84be559624a4586895c0acee1544c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:29.395162 15220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:29.395517 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "61e84be559624a4586895c0acee1544c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "871d0d204466410e8683d77affe62f3a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 33983 } } peers { permanent_uuid: "61e84be559624a4586895c0acee1544c" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "354e9743e7cb4872ac5c3b3fe6e5f8b6" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33227 } attrs { promote: false } } }
I20260619 06:40:29.395769 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354e9743e7cb4872ac5c3b3fe6e5f8b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:40:29.412395 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.402637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:32911 (local address 127.0.0.1:58806)
0619 06:40:29.402794 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.402813 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:29.402913 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:29.404102 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:29.404107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:29.404133 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:29.404505 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:29.404518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.406203 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.406209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:29.408431 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:29.408446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.410963 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.410970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:29.411186 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.411226 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 06:40:29.411294 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:40:29.411438 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 06:40:29.413189 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.403056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:41477 (local address 127.0.0.1:55024)
0619 06:40:29.403199 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.403214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:29.403294 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:29.404530 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:29.404535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:29.404583 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:29.404793 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:29.404803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.406409 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.406415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:29.410721 (+ 4306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:29.410737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.412417 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.412426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:29.412642 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.412679 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:40:29.412742 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 06:40:29.412887 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":2256}
I20260619 06:40:29.414366 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.403310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.7.146.195:32911)
0619 06:40:29.403706 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:29.403712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:29.403729 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:40:29.403794 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:29.403798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:29.403887 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:29.404680 (+ 793us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:29.404858 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.406052 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.409436 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.410696 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.413650 (+ 2954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.413838 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:29.413904 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 06:40:29.414056 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":926,"threads_started":1}
I20260619 06:40:29.414789 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.403697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.7.146.193:41477)
0619 06:40:29.404180 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:29.404188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:29.404207 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:40:29.404282 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:29.404287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:29.404362 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:29.404992 (+ 630us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:29.405154 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.406242 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.410916 (+ 4674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.412181 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.414082 (+ 1901us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.414178 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:29.414227 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 06:40:29.414363 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":473,"threads_started":1}
I20260619 06:40:29.415179 15085 tablet_service.cc:1467] Processing CreateTablet for tablet d48a4c9f863b47ec812e4bf38e7960c5 (DEFAULT_TABLE table=test-table [id=36f45da905b44ba8906dca8449182d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:29.415560 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48a4c9f863b47ec812e4bf38e7960c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:29.415549 14799 tablet_service.cc:1467] Processing CreateTablet for tablet d48a4c9f863b47ec812e4bf38e7960c5 (DEFAULT_TABLE table=test-table [id=36f45da905b44ba8906dca8449182d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:29.415941 14799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48a4c9f863b47ec812e4bf38e7960c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:29.420339 15243 tablet_bootstrap.cc:492] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457: Bootstrap starting.
I20260619 06:40:29.421540 15244 tablet_bootstrap.cc:492] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548: Bootstrap starting.
I20260619 06:40:29.422084 15243 tablet_bootstrap.cc:654] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:29.423478 15244 tablet_bootstrap.cc:654] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:29.425208 15244 tablet_bootstrap.cc:492] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548: No bootstrap required, opened a new log
I20260619 06:40:29.425361 15244 ts_tablet_manager.cc:1397] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:40:29.425800 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.403390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:33669 (local address 127.0.0.1:37850)
0619 06:40:29.412880 (+ 9490us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.412899 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:29.412993 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:29.413694 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:29.413699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:29.413744 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:29.413897 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:29.413908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.421477 (+ 7569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.421483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:29.423299 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:29.423319 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.425047 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.425053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:29.425231 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.425268 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:40:29.425367 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:40:29.425502 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9376,"thread_start_us":11394,"threads_started":1}
I20260619 06:40:29.426446 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.404000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.7.146.194:33669)
0619 06:40:29.413318 (+ 9318us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:29.413323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:29.413341 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:40:29.413403 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:29.413408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:29.413473 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:29.414029 (+ 556us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:29.414177 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.421279 (+ 7102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.423546 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.424849 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.425000 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.425108 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:29.426032 (+ 924us) server_negotiation.cc:299] Negotiation successful
0619 06:40:29.426186 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9191,"thread_start_us":9540,"threads_started":1}
I20260619 06:40:29.426266 15244 raft_consensus.cc:348] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } }
I20260619 06:40:29.426524 15244 raft_consensus.cc:374] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:29.426620 15244 raft_consensus.cc:729] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff7fd3845374e38a40f1eda191ec548, State: Initialized, Role: FOLLOWER
I20260619 06:40:29.426887 15244 consensus_queue.cc:260] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } }
I20260619 06:40:29.427373 14944 tablet_service.cc:1467] Processing CreateTablet for tablet d48a4c9f863b47ec812e4bf38e7960c5 (DEFAULT_TABLE table=test-table [id=36f45da905b44ba8906dca8449182d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:29.427814 14944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48a4c9f863b47ec812e4bf38e7960c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:29.428045 15244 ts_tablet_manager.cc:1428] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:29.430388 15243 tablet_bootstrap.cc:492] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457: No bootstrap required, opened a new log
I20260619 06:40:29.430526 15243 ts_tablet_manager.cc:1397] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 06:40:29.431167 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.429980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.193:59347)
0619 06:40:29.430777 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.430943 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683,"threads_started":1}
I20260619 06:40:29.431298 15243 raft_consensus.cc:348] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } }
I20260619 06:40:29.431562 15243 raft_consensus.cc:374] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:29.431649 15243 raft_consensus.cc:729] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf794b451df4f7a999848e2c06d0457, State: Initialized, Role: FOLLOWER
I20260619 06:40:29.431869 15243 consensus_queue.cc:260] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } }
I20260619 06:40:29.434230 15250 tablet_bootstrap.cc:492] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398: Bootstrap starting.
I20260619 06:40:29.434968 15243 ts_tablet_manager.cc:1428] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260619 06:40:29.435992 15250 tablet_bootstrap.cc:654] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:29.438377 15250 tablet_bootstrap.cc:492] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398: No bootstrap required, opened a new log
I20260619 06:40:29.438514 15250 ts_tablet_manager.cc:1397] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 06:40:29.438680 15152 heartbeater.cc:503] Master 127.7.146.253:35469 was elected leader, sending a full tablet report...
I20260619 06:40:29.439225 15250 raft_consensus.cc:348] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } }
I20260619 06:40:29.439389 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.435473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.195:51053)
0619 06:40:29.437743 (+ 2270us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.439124 (+ 1381us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2129,"thread_start_us":2000,"threads_started":1}
I20260619 06:40:29.439388 15250 raft_consensus.cc:374] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:29.439481 15250 raft_consensus.cc:729] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7022dfc23a67495c9eac0af30f5ce398, State: Initialized, Role: FOLLOWER
I20260619 06:40:29.439683 15250 consensus_queue.cc:260] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } }
I20260619 06:40:29.440325 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.439971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.195:40453)
0619 06:40:29.440110 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.440194 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 06:40:29.440999 15250 ts_tablet_manager.cc:1428] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 06:40:29.442137 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.441268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.194:43615)
0619 06:40:29.441775 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.441910 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":241,"threads_started":1}
I20260619 06:40:29.442705 15011 heartbeater.cc:503] Master 127.7.146.253:35469 was elected leader, sending a full tablet report...
I20260619 06:40:29.444167 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.442719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.194:50875)
0619 06:40:29.443550 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.443909 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":584,"threads_started":1}
I20260619 06:40:29.448055 15253 raft_consensus.cc:482] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:40:29.448232 15253 raft_consensus.cc:504] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } }
I20260619 06:40:29.448891 15253 leader_election.cc:288] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf794b451df4f7a999848e2c06d0457 (127.7.146.195:32911), 8ff7fd3845374e38a40f1eda191ec548 (127.7.146.193:41477)
I20260619 06:40:29.449416 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48a4c9f863b47ec812e4bf38e7960c5" candidate_uuid: "7022dfc23a67495c9eac0af30f5ce398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf794b451df4f7a999848e2c06d0457" is_pre_election: true
I20260619 06:40:29.449443 14819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48a4c9f863b47ec812e4bf38e7960c5" candidate_uuid: "7022dfc23a67495c9eac0af30f5ce398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff7fd3845374e38a40f1eda191ec548" is_pre_election: true
I20260619 06:40:29.449630 15105 raft_consensus.cc:2454] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7022dfc23a67495c9eac0af30f5ce398 in term 0.
I20260619 06:40:29.449658 14819 raft_consensus.cc:2454] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7022dfc23a67495c9eac0af30f5ce398 in term 0.
I20260619 06:40:29.450102 14897 leader_election.cc:302] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7022dfc23a67495c9eac0af30f5ce398, 8ff7fd3845374e38a40f1eda191ec548; no voters:
I20260619 06:40:29.450395 15253 raft_consensus.cc:2781] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:40:29.450537 15253 raft_consensus.cc:482] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:40:29.450627 15253 raft_consensus.cc:3037] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:29.452720 15253 raft_consensus.cc:504] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } }
I20260619 06:40:29.453202 15253 leader_election.cc:288] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [CANDIDATE]: Term 1 election: Requested vote from peers 7bf794b451df4f7a999848e2c06d0457 (127.7.146.195:32911), 8ff7fd3845374e38a40f1eda191ec548 (127.7.146.193:41477)
I20260619 06:40:29.453783 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48a4c9f863b47ec812e4bf38e7960c5" candidate_uuid: "7022dfc23a67495c9eac0af30f5ce398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf794b451df4f7a999848e2c06d0457"
I20260619 06:40:29.453848 14819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48a4c9f863b47ec812e4bf38e7960c5" candidate_uuid: "7022dfc23a67495c9eac0af30f5ce398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff7fd3845374e38a40f1eda191ec548"
I20260619 06:40:29.454020 15105 raft_consensus.cc:3037] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:29.454025 14819 raft_consensus.cc:3037] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:29.456359 14819 raft_consensus.cc:2454] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7022dfc23a67495c9eac0af30f5ce398 in term 1.
I20260619 06:40:29.456373 15105 raft_consensus.cc:2454] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7022dfc23a67495c9eac0af30f5ce398 in term 1.
I20260619 06:40:29.456769 14897 leader_election.cc:302] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7022dfc23a67495c9eac0af30f5ce398, 7bf794b451df4f7a999848e2c06d0457; no voters:
I20260619 06:40:29.457049 15253 raft_consensus.cc:2781] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:29.457440 15253 raft_consensus.cc:686] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 1 LEADER]: Becoming Leader. State: Replica: 7022dfc23a67495c9eac0af30f5ce398, State: Running, Role: LEADER
I20260619 06:40:29.457777 15253 consensus_queue.cc:237] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } } peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } }
I20260619 06:40:29.458350 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.457992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.194:48303)
0619 06:40:29.458147 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.458226 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 06:40:29.460368 14494 catalog_manager.cc:5495] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7022dfc23a67495c9eac0af30f5ce398 (127.7.146.194). New cstate: current_term: 1 leader_uuid: "7022dfc23a67495c9eac0af30f5ce398" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7022dfc23a67495c9eac0af30f5ce398" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 33669 } health_report { overall_health: HEALTHY } } }
I20260619 06:40:29.481860 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.477234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.7.146.194:33669)
0619 06:40:29.477419 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:29.477424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:29.477442 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:40:29.477610 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:29.477615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:29.477712 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:29.477797 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:40:29.478172 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.478968 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.480227 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.480713 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.480823 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.481216 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:40:29.481321 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:40:29.481390 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:29.481522 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 06:40:29.481609 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:40:29.484957 14819 raft_consensus.cc:1261] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548 [term 1 FOLLOWER]: Refusing update from remote peer 7022dfc23a67495c9eac0af30f5ce398: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:29.485109 15105 raft_consensus.cc:1261] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457 [term 1 FOLLOWER]: Refusing update from remote peer 7022dfc23a67495c9eac0af30f5ce398: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:29.485548 15256 consensus_queue.cc:1035] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ff7fd3845374e38a40f1eda191ec548" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 41477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:29.485810 15253 consensus_queue.cc:1035] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf794b451df4f7a999848e2c06d0457" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 32911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:29.487390 14866 heartbeater.cc:503] Master 127.7.146.253:35469 was elected leader, sending a full tablet report...
I20260619 06:40:29.488160 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.487674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.195:59395)
0619 06:40:29.487896 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.487995 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 06:40:29.488323 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.487855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.193:47251)
0619 06:40:29.488078 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.488165 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 06:40:29.493830 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.493292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.194:41351)
0619 06:40:29.493541 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.493651 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260619 06:40:29.494477 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.493076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.195:51201)
0619 06:40:29.494230 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.494318 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1057}
I20260619 06:40:29.494825 15261 mvcc.cc:205] Tried to move back new op lower bound from 7298462635964784640 to 7298462635861413888. Current Snapshot: MvccSnapshot[applied={T|T < 7298462635964784640}]
I20260619 06:40:29.500530 14495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:29.500842 14495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:29.502185 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.501409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.194:50141)
0619 06:40:29.501961 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.502037 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470}
I20260619 06:40:29.504807 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.504336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:33983 (local address 127.7.146.193:37923)
0619 06:40:29.504476 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:29.504608 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 06:40:29.506966 14495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61e84be559624a4586895c0acee1544c: Sending DeleteTablet for 3 replicas of tablet d48a4c9f863b47ec812e4bf38e7960c5
I20260619 06:40:29.507875 14799 tablet_service.cc:1514] Processing DeleteTablet for tablet d48a4c9f863b47ec812e4bf38e7960c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:29 UTC) from {username='oryx', principal='oryx/127.7.146.253@KRBTEST.COM'} at 127.0.0.1:55024
I20260619 06:40:29.508044 14944 tablet_service.cc:1514] Processing DeleteTablet for tablet d48a4c9f863b47ec812e4bf38e7960c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:29 UTC) from {username='oryx', principal='oryx/127.7.146.253@KRBTEST.COM'} at 127.0.0.1:37850
I20260619 06:40:29.508103 14798 tablet_service.cc:1514] Processing DeleteTablet for tablet d48a4c9f863b47ec812e4bf38e7960c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:29 UTC) from {username='oryx', principal='oryx/127.7.146.253@KRBTEST.COM'} at 127.0.0.1:55024
I20260619 06:40:29.508415 14943 tablet_service.cc:1514] Processing DeleteTablet for tablet d48a4c9f863b47ec812e4bf38e7960c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:29 UTC) from {username='oryx', principal='oryx/127.7.146.253@KRBTEST.COM'} at 127.0.0.1:37850
I20260619 06:40:29.508641 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet d48a4c9f863b47ec812e4bf38e7960c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:29 UTC) from {username='oryx', principal='oryx/127.7.146.253@KRBTEST.COM'} at 127.0.0.1:58806
I20260619 06:40:29.509385 15268 tablet_replica.cc:330] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398: stopping tablet replica
I20260619 06:40:29.509590 15268 raft_consensus.cc:2229] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:40:29.509948 15268 raft_consensus.cc:2258] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:29.510514 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 14664
I20260619 06:40:29.513017 15269 tablet_replica.cc:330] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457: stopping tablet replica
I20260619 06:40:29.513221 15269 raft_consensus.cc:2229] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:29.513511 15269 raft_consensus.cc:2258] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:29.514029 15267 tablet_replica.cc:330] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548: stopping tablet replica
I20260619 06:40:29.514186 15267 raft_consensus.cc:2229] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:29.514434 15267 raft_consensus.cc:2258] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:29.514775 15269 ts_tablet_manager.cc:1905] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:29.515204 15267 ts_tablet_manager.cc:1905] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:29.517938 15268 ts_tablet_manager.cc:1905] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:29.519671 15267 ts_tablet_manager.cc:1918] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:29.519783 15267 log.cc:1196] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/d48a4c9f863b47ec812e4bf38e7960c5
I20260619 06:40:29.520164 15267 ts_tablet_manager.cc:1939] T d48a4c9f863b47ec812e4bf38e7960c5 P 8ff7fd3845374e38a40f1eda191ec548: Deleting consensus metadata
I20260619 06:40:29.521171 14479 catalog_manager.cc:4841] TS 8ff7fd3845374e38a40f1eda191ec548 (127.7.146.193:41477): tablet d48a4c9f863b47ec812e4bf38e7960c5 (table test-table [id=36f45da905b44ba8906dca8449182d82]) successfully deleted
I20260619 06:40:29.521262 15269 ts_tablet_manager.cc:1918] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:29.521399 15269 log.cc:1196] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/d48a4c9f863b47ec812e4bf38e7960c5
W20260619 06:40:29.521730 14479 catalog_manager.cc:4808] TS 8ff7fd3845374e38a40f1eda191ec548 (127.7.146.193:41477): delete failed for tablet d48a4c9f863b47ec812e4bf38e7960c5 because the tablet was not found. No further retry: Not found: Tablet not found: d48a4c9f863b47ec812e4bf38e7960c5
I20260619 06:40:29.521818 15269 ts_tablet_manager.cc:1939] T d48a4c9f863b47ec812e4bf38e7960c5 P 7bf794b451df4f7a999848e2c06d0457: Deleting consensus metadata
I20260619 06:40:29.522827 14479 catalog_manager.cc:4841] TS 7bf794b451df4f7a999848e2c06d0457 (127.7.146.195:32911): tablet d48a4c9f863b47ec812e4bf38e7960c5 (table test-table [id=36f45da905b44ba8906dca8449182d82]) successfully deleted
I20260619 06:40:29.526319 15268 ts_tablet_manager.cc:1918] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:29.526422 15268 log.cc:1196] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/d48a4c9f863b47ec812e4bf38e7960c5
I20260619 06:40:29.526803 15268 ts_tablet_manager.cc:1939] T d48a4c9f863b47ec812e4bf38e7960c5 P 7022dfc23a67495c9eac0af30f5ce398: Deleting consensus metadata
I20260619 06:40:29.527668 14481 catalog_manager.cc:4841] TS 7022dfc23a67495c9eac0af30f5ce398 (127.7.146.194:33669): tablet d48a4c9f863b47ec812e4bf38e7960c5 (table test-table [id=36f45da905b44ba8906dca8449182d82]) successfully deleted
W20260619 06:40:29.527990 14481 catalog_manager.cc:4808] TS 7022dfc23a67495c9eac0af30f5ce398 (127.7.146.194:33669): delete failed for tablet d48a4c9f863b47ec812e4bf38e7960c5 because the tablet was not found. No further retry: Not found: Tablet not found: d48a4c9f863b47ec812e4bf38e7960c5
I20260619 06:40:29.539141 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.511837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.7.146.193:41477)
0619 06:40:29.511991 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:29.511997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:29.526956 (+ 14959us) server_negotiation.cc:407] Connection header received
0619 06:40:29.527153 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:29.527157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:29.527226 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:29.527303 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:29.528801 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.529469 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.530618 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.531028 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.531129 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.533524 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:29.533537 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:29.533553 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:29.533583 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:29.536389 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:29.537111 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:29.537118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:29.537120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:29.537259 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:29.537610 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:29.537615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:29.537618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:29.538037 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:29.538293 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:29.538608 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 06:40:29.538810 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 06:40:29.539827 14859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:29.630609 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 14664
I20260619 06:40:29.644598 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 14876
I20260619 06:40:29.658025 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.645318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.7.146.194:33669)
0619 06:40:29.645543 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:29.645549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:29.645594 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:40:29.645823 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:29.645829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:29.645911 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:29.646015 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:29.647425 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.648262 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.649601 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.650037 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.650162 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.652789 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:29.652804 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:29.652819 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:29.652851 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:29.655377 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:29.656038 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:29.656045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:29.656047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:29.656175 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:29.656543 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:29.656547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:29.656549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:29.656930 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:29.657210 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:29.657461 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 06:40:29.657665 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 06:40:29.658691 15004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:29.775703 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 14876
I20260619 06:40:29.792649 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 15022
I20260619 06:40:29.806041 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:29.793436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.7.146.195:32911)
0619 06:40:29.793601 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:29.793606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:29.793624 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:40:29.793727 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:29.793730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:29.793801 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:29.793898 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:29.795193 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.795956 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.797294 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:29.797733 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:29.797876 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:29.800312 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:29.800327 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:29.800363 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:29.800402 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:29.803286 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:29.803954 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:29.803964 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:29.803968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:29.804109 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:29.804458 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:29.804461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:29.804463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:29.804946 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:29.805158 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:29.805539 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 06:40:29.805731 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:40:29.806700 15145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:29.898756 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 15022
I20260619 06:40:29.912500 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 14359
I20260619 06:40:29.913616 14525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:30.033789 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 14359
I20260619 06:40:30.049883 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 14459
I20260619 06:40:30.050916 14715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:30.163964 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 14459
2026-06-19T06:40:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8961 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:40:30 dist-test-slave-qfk4 krb5kdc[15278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:40:30 dist-test-slave-qfk4 krb5kdc[15278](info): set up 2 sockets
Jun 19 06:40:30 dist-test-slave-qfk4 krb5kdc[15278](info): commencing operation
krb5kdc: starting...
W20260619 06:40:31.169358 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:40:31 dist-test-slave-qfk4 krb5kdc[15278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:40:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.254.keytab.
Entry for principal oryx/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.254.keytab.
I20260619 06:40:31.371626 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34441
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.254.keytab
--principal=oryx/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:34441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:31.686525 15291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:31.686826 15291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:31.686920 15291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:31.694263 15291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:31.694375 15291 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:40:31.694422 15291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:31.694504 15291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:31.694566 15291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:31.694639 15291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:31.704545 15291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:34441
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.254.keytab
--principal=oryx/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:31.705935 15291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:31.707324 15291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:31.718533 15299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:31.718683 15297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:31.718660 15296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:31.719718 15291 server_base.cc:1034] running on GCE node
I20260619 06:40:31.720479 15291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:31.721954 15291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:31.723181 15291 hybrid_clock.cc:648] HybridClock initialized: now 1781851231723113 us; error 37 us; skew 500 ppm
I20260619 06:40:31.723489 15291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:31 dist-test-slave-qfk4 krb5kdc[15278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851231, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:31.726540 15291 init.cc:377] Logged in from keytab as oryx/127.7.146.254@KRBTEST.COM (short username oryx)
I20260619 06:40:31.727955 15291 webserver.cc:460] Webserver started at http://127.7.146.254:34983/ using document root <none> and password file <none>
I20260619 06:40:31.728688 15291 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:31.728789 15291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:31.729099 15291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:31.732008 15291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "0d2effef6fef431c9369a36274396f49"
format_stamp: "Formatted at 2026-06-19 06:40:31 on dist-test-slave-qfk4"
server_key: "d35d6a29077a846987a05d0a9845ca36"
server_key_iv: "7dbe1b6d13793b41594d351a1707b4ad"
server_key_version: "encryptionkey@0"
I20260619 06:40:31.732810 15291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "0d2effef6fef431c9369a36274396f49"
format_stamp: "Formatted at 2026-06-19 06:40:31 on dist-test-slave-qfk4"
server_key: "d35d6a29077a846987a05d0a9845ca36"
server_key_iv: "7dbe1b6d13793b41594d351a1707b4ad"
server_key_version: "encryptionkey@0"
I20260619 06:40:31.737812 15291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:40:31.741335 15306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:31.742729 15291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:40:31.742904 15291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "0d2effef6fef431c9369a36274396f49"
format_stamp: "Formatted at 2026-06-19 06:40:31 on dist-test-slave-qfk4"
server_key: "d35d6a29077a846987a05d0a9845ca36"
server_key_iv: "7dbe1b6d13793b41594d351a1707b4ad"
server_key_version: "encryptionkey@0"
I20260619 06:40:31.743162 15291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:31.767830 15291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:31.772063 15291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:31.772318 15291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:31.792078 15291 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:34441
I20260619 06:40:31.792155 15367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:34441 every 8 connection(s)
I20260619 06:40:31.793462 15291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:31.797277 15368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:31.800782 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 15291
I20260619 06:40:31.801004 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:31.801789 7755 external_mini_cluster.cc:1428] Setting key f97740032d50ae43ad8a7720b26fe01c
I20260619 06:40:31.810527 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49: Bootstrap starting.
I20260619 06:40:31.814324 15368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:31.815693 15368 log.cc:824] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:40:31 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:31.819027 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49: No bootstrap required, opened a new log
I20260619 06:40:31.825599 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:31.803960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.7.146.254:34441)
0619 06:40:31.804790 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:31.804804 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:31.804846 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:40:31.806508 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:31.806533 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:31.806986 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:31.807336 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:31.809068 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:31.810404 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:31.812101 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:31.812945 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:31.813090 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:31.817956 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:31.817966 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:31.817982 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:31.818024 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:31.821141 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:31.822027 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:31.822035 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:31.822042 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:31.822219 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:31.822654 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:31.822659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:31.822661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:31.823293 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:31.823519 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:31.824071 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 06:40:31.824470 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":341,"threads_started":1}
I20260619 06:40:31.828008 15368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2effef6fef431c9369a36274396f49" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34441 } }
I20260619 06:40:31.828387 15368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:31.828503 15368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2effef6fef431c9369a36274396f49, State: Initialized, Role: FOLLOWER
I20260619 06:40:31.829074 15368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2effef6fef431c9369a36274396f49" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34441 } }
I20260619 06:40:31.829301 15368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:31.829417 15368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:31.829572 15368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:31.831636 15368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2effef6fef431c9369a36274396f49" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34441 } }
I20260619 06:40:31.832127 15368 leader_election.cc:302] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2effef6fef431c9369a36274396f49; no voters:
I20260619 06:40:31.832674 15368 leader_election.cc:288] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:31.832777 15373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:31.833429 15373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [term 1 LEADER]: Becoming Leader. State: Replica: 0d2effef6fef431c9369a36274396f49, State: Running, Role: LEADER
I20260619 06:40:31.833973 15373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2effef6fef431c9369a36274396f49" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34441 } }
I20260619 06:40:31.834587 15368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:31.836777 15374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d2effef6fef431c9369a36274396f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2effef6fef431c9369a36274396f49" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34441 } } }
I20260619 06:40:31.836617 15375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d2effef6fef431c9369a36274396f49. Latest consensus state: current_term: 1 leader_uuid: "0d2effef6fef431c9369a36274396f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2effef6fef431c9369a36274396f49" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34441 } } }
I20260619 06:40:31.837265 15375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:31.838374 15374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:31.845192 15379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:31.850392 15379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:31.858970 15379 catalog_manager.cc:1347] Generated new cluster ID: 044cf23d460a4d148b2e8bd538f4857e
I20260619 06:40:31.859094 15379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:31.901674 15379 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:31.903559 15379 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:31.919267 15379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49: Generated new TSK 0
I20260619 06:40:31.920377 15379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab.
Entry for principal oryx/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab.
I20260619 06:40:31.984647 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34441
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab
--principal=oryx/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:32.314883 15394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:32.315259 15394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:32.315454 15394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:32.324257 15394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:32.324378 15394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:32.324501 15394 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:40:32.324618 15394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:40:32.334707 15394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab
--principal=oryx/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.146.254:34441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:32.336098 15394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:32.337515 15394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:32.350034 15399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:32.350080 15402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:32.350111 15400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:32.351203 15394 server_base.cc:1034] running on GCE node
I20260619 06:40:32.351792 15394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:32.352912 15394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:32.354161 15394 hybrid_clock.cc:648] HybridClock initialized: now 1781851232354095 us; error 43 us; skew 500 ppm
I20260619 06:40:32.354475 15394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:32 dist-test-slave-qfk4 krb5kdc[15278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851232, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:32.357649 15394 init.cc:377] Logged in from keytab as oryx/127.7.146.193@KRBTEST.COM (short username oryx)
I20260619 06:40:32.359128 15394 webserver.cc:460] Webserver started at http://127.7.146.193:41873/ using document root <none> and password file <none>
I20260619 06:40:32.359798 15394 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:32.359989 15394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:32.360337 15394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:32.363302 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3"
format_stamp: "Formatted at 2026-06-19 06:40:32 on dist-test-slave-qfk4"
server_key: "4b83dcbe7615ff4517cdad13183e2e04"
server_key_iv: "be784e68f16de0db1a2da3dd262157d2"
server_key_version: "encryptionkey@0"
I20260619 06:40:32.364087 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3"
format_stamp: "Formatted at 2026-06-19 06:40:32 on dist-test-slave-qfk4"
server_key: "4b83dcbe7615ff4517cdad13183e2e04"
server_key_iv: "be784e68f16de0db1a2da3dd262157d2"
server_key_version: "encryptionkey@0"
I20260619 06:40:32.369307 15394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 06:40:32.372947 15409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:32.374275 15394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:40:32.374428 15394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3"
format_stamp: "Formatted at 2026-06-19 06:40:32 on dist-test-slave-qfk4"
server_key: "4b83dcbe7615ff4517cdad13183e2e04"
server_key_iv: "be784e68f16de0db1a2da3dd262157d2"
server_key_version: "encryptionkey@0"
I20260619 06:40:32.374629 15394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:32.394945 15394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:32.399392 15394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:32.399655 15394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:32.402415 15394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:32.402501 15394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:32.402617 15394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:32.402671 15394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:40:32 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851232, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:32.426743 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:32.403183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:41447 (local address 127.7.146.254:34441)
0619 06:40:32.403633 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:32.403640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:32.404981 (+ 1341us) server_negotiation.cc:407] Connection header received
0619 06:40:32.407653 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:32.407659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:32.407751 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:32.407874 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:32.410379 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.411380 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.413652 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.414236 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.414371 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:32.419660 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:32.419667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:32.419672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:32.419725 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:32.423173 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:32.424191 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:32.424197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:32.424200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:32.424319 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:32.424836 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:32.424840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:32.424842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:32.425123 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:32.425251 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:32.426151 (+ 900us) server_negotiation.cc:299] Negotiation successful
0619 06:40:32.426397 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":220,"threads_started":1}
I20260619 06:40:32.428352 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:32.403890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34441 (local address 127.7.146.193:41447)
0619 06:40:32.404796 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:32.404866 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:32.407311 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:32.408364 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:32.408380 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:32.409205 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:32.410140 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:32.410170 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.411545 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.411549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:32.413510 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:32.413522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.414519 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.414527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:32.414875 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:32.415757 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:32.415800 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:32.419372 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:32.423367 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:32.423377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:32.423399 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:32.424058 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:32.424461 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:32.424466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:32.424471 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:32.424722 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:32.425248 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:32.425256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:32.425724 (+ 468us) client_negotiation.cc:769] Sending connection context
0619 06:40:32.426048 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 06:40:32.426627 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":383,"threads_started":1}
I20260619 06:40:32.438442 15394 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:36523
I20260619 06:40:32.438527 15529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:36523 every 8 connection(s)
I20260619 06:40:32.439774 15394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:40:32.442013 15530 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34441
I20260619 06:40:32.442234 15530 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:32.442947 15530 heartbeater.cc:511] Master 127.7.146.254:34441 requested a full tablet report, sending...
I20260619 06:40:32.445400 15323 ts_manager.cc:194] Registered new tserver with Master: 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193:36523)
I20260619 06:40:32.447253 15323 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:41447
I20260619 06:40:32.448686 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:32.435242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34441 (local address 127.7.146.193:44447)
0619 06:40:32.435942 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:32.435966 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:32.436132 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:32.436514 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:32.436519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:32.437002 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:32.437273 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:32.437285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.438693 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.438699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:32.439740 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:32.439753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.440953 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.440960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:32.441158 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:32.441732 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:32.441749 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:32.442602 (+ 853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:32.445931 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:32.445938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:32.445942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:32.446458 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:32.446924 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:32.446929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:32.446933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:32.447082 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:32.447614 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:32.447619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:32.447808 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 06:40:32.447975 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 06:40:32.448288 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":425,"threads_started":1}
I20260619 06:40:32.448725 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:32.435622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:44447 (local address 127.7.146.254:34441)
0619 06:40:32.435775 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:32.435780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:32.436068 (+ 288us) server_negotiation.cc:407] Connection header received
0619 06:40:32.436288 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:32.436294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:32.436387 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:32.436482 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:32.437479 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.438546 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.439891 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.440491 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.440714 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:32.442778 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:32.442785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:32.442788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:32.442819 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:32.445727 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:32.446595 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:32.446599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:32.446602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:32.446759 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:32.447190 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:32.447194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:32.447196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:32.447504 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:32.447613 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:32.448111 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 06:40:32.448361 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 06:40:32.449088 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 15394
I20260619 06:40:32.449280 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:40:32.449913 7755 external_mini_cluster.cc:1428] Setting key 61a9f6945c3fd56f3de787393214042e
WARNING: no policy specified for oryx/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab.
Entry for principal oryx/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab.
I20260619 06:40:32.495260 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34441
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab
--principal=oryx/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:32.817335 15534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:32.817611 15534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:32.817726 15534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:32.825008 15534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:32.825129 15534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:32.825251 15534 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:40:32.825354 15534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:40:32.835372 15534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab
--principal=oryx/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.146.254:34441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:32.836719 15534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:32.838038 15534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:32.851331 15539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:32.851361 15542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:32.851408 15540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:32.851925 15534 server_base.cc:1034] running on GCE node
I20260619 06:40:32.852617 15534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:32.853675 15534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:32.854961 15534 hybrid_clock.cc:648] HybridClock initialized: now 1781851232854883 us; error 48 us; skew 500 ppm
I20260619 06:40:32.855269 15534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:32 dist-test-slave-qfk4 krb5kdc[15278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851232, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:32.862556 15534 init.cc:377] Logged in from keytab as oryx/127.7.146.194@KRBTEST.COM (short username oryx)
I20260619 06:40:32.864104 15534 webserver.cc:460] Webserver started at http://127.7.146.194:35115/ using document root <none> and password file <none>
I20260619 06:40:32.864851 15534 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:32.864949 15534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:32.865218 15534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:32.868038 15534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff"
format_stamp: "Formatted at 2026-06-19 06:40:32 on dist-test-slave-qfk4"
server_key: "c8662395b0da9d1eae1451aaeaea7eb2"
server_key_iv: "c4efd6505d9739bcc7db4099a4f07ebd"
server_key_version: "encryptionkey@0"
I20260619 06:40:32.868800 15534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff"
format_stamp: "Formatted at 2026-06-19 06:40:32 on dist-test-slave-qfk4"
server_key: "c8662395b0da9d1eae1451aaeaea7eb2"
server_key_iv: "c4efd6505d9739bcc7db4099a4f07ebd"
server_key_version: "encryptionkey@0"
I20260619 06:40:32.873767 15534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:40:32.877118 15549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:32.878522 15534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 06:40:32.878669 15534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff"
format_stamp: "Formatted at 2026-06-19 06:40:32 on dist-test-slave-qfk4"
server_key: "c8662395b0da9d1eae1451aaeaea7eb2"
server_key_iv: "c4efd6505d9739bcc7db4099a4f07ebd"
server_key_version: "encryptionkey@0"
I20260619 06:40:32.878867 15534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:32.899251 15534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:32.903393 15534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:32.903637 15534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:32.906343 15534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:32.906422 15534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:32.906544 15534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:32.906596 15534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:40:32 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851232, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:32.930516 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:32.907067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:38025 (local address 127.7.146.254:34441)
0619 06:40:32.907253 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:32.907258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:32.908760 (+ 1502us) server_negotiation.cc:407] Connection header received
0619 06:40:32.910762 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:32.910768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:32.910857 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:32.910945 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:32.913110 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.914095 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.916418 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.916921 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.917045 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:32.922086 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:32.922091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:32.922094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:32.922128 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:32.925451 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:32.926397 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:32.926403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:32.926407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:32.926568 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:32.927144 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:32.927150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:32.927153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:32.927472 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:32.927609 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:32.929992 (+ 2383us) server_negotiation.cc:299] Negotiation successful
0619 06:40:32.930208 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:40:32.931679 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:32.907690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34441 (local address 127.7.146.194:38025)
0619 06:40:32.908551 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:32.908625 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:32.910451 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:32.911227 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:32.911240 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:32.912011 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:32.912917 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:32.912951 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.914289 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.914294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:32.916265 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:32.916281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.917174 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.917183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:32.917535 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:32.918386 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:32.918425 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:32.921863 (+ 3438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:32.925586 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:32.925596 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:32.925615 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:32.926231 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:32.926705 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:32.926709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:32.926711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:32.926996 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:32.927689 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:32.927697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:32.928188 (+ 491us) client_negotiation.cc:769] Sending connection context
0619 06:40:32.928511 (+ 323us) client_negotiation.cc:240] Negotiation successful
0619 06:40:32.929093 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":350,"threads_started":1}
I20260619 06:40:32.943552 15534 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:42797
I20260619 06:40:32.943578 15668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:42797 every 8 connection(s)
I20260619 06:40:32.944988 15534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:40:32.947662 15669 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34441
I20260619 06:40:32.947911 15669 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:32.948683 15669 heartbeater.cc:511] Master 127.7.146.254:34441 requested a full tablet report, sending...
I20260619 06:40:32.950143 15322 ts_manager.cc:194] Registered new tserver with Master: 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797)
I20260619 06:40:32.951112 15322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:38025
I20260619 06:40:32.953084 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:32.938027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:50063 (local address 127.7.146.254:34441)
0619 06:40:32.938217 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:32.938222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:32.939032 (+ 810us) server_negotiation.cc:407] Connection header received
0619 06:40:32.939240 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:32.939246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:32.939334 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:32.939435 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:32.941167 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.942233 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.944237 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.944917 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.945031 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:32.947441 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:32.947447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:32.947452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:32.947511 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:32.950423 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:32.951066 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:32.951072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:32.951075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:32.951262 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:32.951656 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:32.951662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:32.951665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:32.952037 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:32.952145 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:32.952594 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 06:40:32.952795 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:40:32.953475 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:32.938177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34441 (local address 127.7.146.194:50063)
0619 06:40:32.938885 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:32.938909 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:32.939098 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:32.939642 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:32.939648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:32.940375 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:32.940940 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:32.940959 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.942417 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.942423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:32.944072 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:32.944087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:32.945115 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:32.945121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:32.945300 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:32.946010 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:32.946037 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:32.947265 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:32.950570 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:32.950575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:32.950577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:32.950951 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:32.951371 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:32.951375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:32.951378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:32.951544 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:32.952166 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:32.952171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:32.952349 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 06:40:32.952502 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 06:40:32.952939 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":103,"thread_start_us":394,"threads_started":1}
I20260619 06:40:32.955219 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 15534
I20260619 06:40:32.955366 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:40:32.955777 7755 external_mini_cluster.cc:1428] Setting key e24c09bf9af0b734843e7b80c0c05498
WARNING: no policy specified for oryx/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab.
Entry for principal oryx/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab.
I20260619 06:40:32.998032 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34441
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab
--principal=oryx/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:33.313058 15673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:33.313341 15673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:33.313437 15673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:33.320677 15673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:33.320789 15673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:33.320879 15673 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:40:33.320960 15673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:40:33.330729 15673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab
--principal=oryx/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.146.254:34441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:33.332019 15673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:33.333266 15673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:33.346071 15681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:33.346212 15678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:33.346117 15679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:33.347028 15673 server_base.cc:1034] running on GCE node
I20260619 06:40:33.347653 15673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:33.348634 15673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:33.349838 15673 hybrid_clock.cc:648] HybridClock initialized: now 1781851233349771 us; error 47 us; skew 500 ppm
I20260619 06:40:33.350116 15673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:33 dist-test-slave-qfk4 krb5kdc[15278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851233, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:33.353099 15673 init.cc:377] Logged in from keytab as oryx/127.7.146.195@KRBTEST.COM (short username oryx)
I20260619 06:40:33.354518 15673 webserver.cc:460] Webserver started at http://127.7.146.195:44965/ using document root <none> and password file <none>
I20260619 06:40:33.355167 15673 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:33.355261 15673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:33.355566 15673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:33.358412 15673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "b0240319ecc14b0cb857acff473aa64d"
format_stamp: "Formatted at 2026-06-19 06:40:33 on dist-test-slave-qfk4"
server_key: "828fb89b75902acf01abe53e7699f8c5"
server_key_iv: "22575b3dba2ad8b1e2d483436662cf71"
server_key_version: "encryptionkey@0"
I20260619 06:40:33.359146 15673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "b0240319ecc14b0cb857acff473aa64d"
format_stamp: "Formatted at 2026-06-19 06:40:33 on dist-test-slave-qfk4"
server_key: "828fb89b75902acf01abe53e7699f8c5"
server_key_iv: "22575b3dba2ad8b1e2d483436662cf71"
server_key_version: "encryptionkey@0"
I20260619 06:40:33.364130 15673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:40:33.367595 15688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:33.368906 15673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:33.369062 15673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "b0240319ecc14b0cb857acff473aa64d"
format_stamp: "Formatted at 2026-06-19 06:40:33 on dist-test-slave-qfk4"
server_key: "828fb89b75902acf01abe53e7699f8c5"
server_key_iv: "22575b3dba2ad8b1e2d483436662cf71"
server_key_version: "encryptionkey@0"
I20260619 06:40:33.369261 15673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:33.392016 15673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:33.396124 15673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:33.396380 15673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:33.398942 15673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:33.399027 15673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:33.399155 15673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:33.399215 15673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:40:33 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851233, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:33.424276 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:33.400323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34441 (local address 127.7.146.195:59869)
0619 06:40:33.401197 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:33.401244 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:33.403753 (+ 2509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:33.404713 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:33.404727 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:33.405600 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:33.406578 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:33.406607 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:33.408056 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:33.408061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:33.410159 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:33.410173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:33.411175 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:33.411183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:33.411486 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:33.412346 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:33.412384 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:33.415686 (+ 3302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:33.419556 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:33.419565 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:33.419588 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:33.420386 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:33.420878 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:33.420884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:33.420888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:33.421110 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:33.421727 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:33.421735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:33.422278 (+ 543us) client_negotiation.cc:769] Sending connection context
0619 06:40:33.422651 (+ 373us) client_negotiation.cc:240] Negotiation successful
0619 06:40:33.423106 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":348,"threads_started":1}
I20260619 06:40:33.425267 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:33.399711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:59869 (local address 127.7.146.254:34441)
0619 06:40:33.399929 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:33.399936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:33.401380 (+ 1444us) server_negotiation.cc:407] Connection header received
0619 06:40:33.404119 (+ 2739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:33.404127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:33.404238 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:33.404386 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:33.406797 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:33.407842 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:33.410317 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:33.410910 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:33.411061 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:33.415979 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:33.415987 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:33.415993 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:33.416039 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:33.419399 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:33.420536 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:33.420542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:33.420559 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:33.420767 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:33.421225 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:33.421230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:33.421234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:33.421586 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:33.421726 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:33.424867 (+ 3141us) server_negotiation.cc:299] Negotiation successful
0619 06:40:33.425043 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:40:33.435573 15673 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:35333
I20260619 06:40:33.435587 15807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:35333 every 8 connection(s)
I20260619 06:40:33.437286 15673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:40:33.438082 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 15673
I20260619 06:40:33.438238 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:40:33.438721 7755 external_mini_cluster.cc:1428] Setting key a8a592b15fba00e52b81cf145cb3d2ef
I20260619 06:40:33.445689 15808 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34441
I20260619 06:40:33.446062 15808 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:33.446774 15808 heartbeater.cc:511] Master 127.7.146.254:34441 requested a full tablet report, sending...
I20260619 06:40:33.448237 15322 ts_manager.cc:194] Registered new tserver with Master: b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333)
I20260619 06:40:33.449352 15322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:59869
I20260619 06:40:33.451364 15530 heartbeater.cc:503] Master 127.7.146.254:34441 was elected leader, sending a full tablet report...
I20260619 06:40:33.454499 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:40:33.457281 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:33.431567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34441 (local address 127.7.146.195:36169)
0619 06:40:33.443809 (+ 12242us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:33.443832 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:33.444011 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:33.444377 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:33.444382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:33.445469 (+ 1087us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:33.445957 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:33.445971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:33.447046 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:33.447051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:33.448384 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:33.448399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:33.449415 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:33.449424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:33.449591 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:33.450317 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:33.450341 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:33.451385 (+ 1044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:33.454972 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:33.454976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:33.454985 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:33.455359 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:33.455712 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:33.455716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:33.455718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:33.455829 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:33.456341 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:33.456345 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:33.456493 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 06:40:33.456665 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 06:40:33.456880 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12052,"thread_start_us":12541,"threads_started":1}
I20260619 06:40:33.457283 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:33.431973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:36169 (local address 127.7.146.254:34441)
0619 06:40:33.432130 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:33.432136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:33.443935 (+ 11799us) server_negotiation.cc:407] Connection header received
0619 06:40:33.444135 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:33.444139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:33.444213 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:33.444309 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:33.446107 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:33.446890 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:33.448502 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:33.449211 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:33.449344 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:33.451602 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:33.451609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:33.451614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:33.451654 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:33.454854 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:33.455472 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:33.455477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:33.455481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:33.455626 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:33.455907 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:33.455912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:33.455915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:33.456235 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:33.456385 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:33.456770 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 06:40:33.456963 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 06:40:33 dist-test-slave-qfk4 krb5kdc[15278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:40:33 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:33 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:33.954669 15669 heartbeater.cc:503] Master 127.7.146.254:34441 was elected leader, sending a full tablet report...
I20260619 06:40:34.452834 15808 heartbeater.cc:503] Master 127.7.146.254:34441 was elected leader, sending a full tablet report...
W20260619 06:40:34.456512 15816 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.960s user 0.000s sys 0.012s
W20260619 06:40:34.457485 15418 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:33.490903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59750 (local address 127.7.146.254:34441)
0619 06:40:33.491121 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:33.491128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:33.491309 (+ 181us) server_negotiation.cc:407] Connection header received
0619 06:40:33.491498 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:33.491505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:33.491569 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:33.491656 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:33.493065 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:33.493802 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:33.495023 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:33.495414 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:33.495523 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.457271 (+961748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59750: BlockingRecv error: recv got EOF from 127.0.0.1:59750 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 06:40:34 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:34.473686 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.460749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.7.146.254:34441)
0619 06:40:34.460911 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:34.460917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:34.461227 (+ 310us) server_negotiation.cc:407] Connection header received
0619 06:40:34.461392 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:34.461398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:34.461477 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:34.461559 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:34.462761 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.463481 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.464656 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.465093 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.465213 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.468870 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:34.468875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:34.468878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:34.468918 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:34.471221 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:34.471866 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:34.471871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:34.471874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:34.472011 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:34.472294 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:34.472297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:34.472299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:34.472587 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:34.472814 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:34.473261 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 06:40:34.473453 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:40:34.477597 15322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:40:34.487131 15322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:40:34.529213 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.499281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.7.146.193:36523)
0619 06:40:34.520666 (+ 21385us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:34.520678 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:34.521441 (+ 763us) server_negotiation.cc:407] Connection header received
0619 06:40:34.521567 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:34.521575 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:34.521860 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:34.522074 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:34.522966 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.524305 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.526442 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.527672 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.527791 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.528026 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:34.528332 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 06:40:34.528519 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21184,"thread_start_us":21030,"threads_started":1}
I20260619 06:40:34.529860 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.498759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:36523 (local address 127.0.0.1:44242)
0619 06:40:34.521141 (+ 22382us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:34.521164 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:34.521302 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:34.522344 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:34.522359 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:34.522397 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:34.522769 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.522779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.524418 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.524423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.526267 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.526283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.527870 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.527875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.528106 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.528159 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 06:40:34.528283 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 06:40:34.528481 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22253,"thread_start_us":22503,"threads_started":1}
I20260619 06:40:34.530174 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.499169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:35333 (local address 127.0.0.1:50674)
0619 06:40:34.521527 (+ 22358us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:34.521547 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:34.521777 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:34.522697 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:34.522701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:34.522729 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:34.523164 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.523186 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.524854 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.524859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.526827 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.526843 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.529074 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.529081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.529239 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.529271 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:40:34.529573 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 06:40:34.529716 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22245,"thread_start_us":21670,"threads_started":1}
I20260619 06:40:34.530627 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.498623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:42797 (local address 127.0.0.1:52848)
0619 06:40:34.519123 (+ 20500us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:34.519172 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:34.519362 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:34.523857 (+ 4495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:34.523862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:34.523886 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:34.524199 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.524209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.525969 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.525974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.527772 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.527785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.529748 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.529753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.530021 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.530053 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:40:34.530115 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 06:40:34.530318 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20193,"thread_start_us":20931,"threads_started":1}
I20260619 06:40:34.530762 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.499714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.7.146.195:35333)
0619 06:40:34.512906 (+ 13192us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:34.512916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:34.521672 (+ 8756us) server_negotiation.cc:407] Connection header received
0619 06:40:34.521969 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:34.521978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:34.522228 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:34.522425 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:34.523373 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.524731 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.526995 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.528609 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.528734 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.528927 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:34.529948 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0619 06:40:34.530503 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12945,"thread_start_us":12780,"threads_started":1}
I20260619 06:40:34.531565 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.498850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.7.146.194:42797)
0619 06:40:34.523339 (+ 24489us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:34.523348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:34.523374 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:40:34.523470 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:34.523476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:34.523674 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:34.523842 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:34.524368 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.525831 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.527974 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.529538 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.530863 (+ 1325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.531079 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:34.531146 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 06:40:34.531303 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24336,"thread_start_us":24198,"threads_started":1}
I20260619 06:40:34.534029 15738 tablet_service.cc:1467] Processing CreateTablet for tablet db10f126476f4189a8bc70d44597de23 (DEFAULT_TABLE table=test-table [id=514d1a5c14a84dd4aa920398dc3041ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:34.534101 15460 tablet_service.cc:1467] Processing CreateTablet for tablet db10f126476f4189a8bc70d44597de23 (DEFAULT_TABLE table=test-table [id=514d1a5c14a84dd4aa920398dc3041ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:34.534838 15599 tablet_service.cc:1467] Processing CreateTablet for tablet db10f126476f4189a8bc70d44597de23 (DEFAULT_TABLE table=test-table [id=514d1a5c14a84dd4aa920398dc3041ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:34.535398 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db10f126476f4189a8bc70d44597de23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:34.535490 15460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db10f126476f4189a8bc70d44597de23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:34.536074 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db10f126476f4189a8bc70d44597de23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:34.545850 15833 tablet_bootstrap.cc:492] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d: Bootstrap starting.
I20260619 06:40:34.549111 15833 tablet_bootstrap.cc:654] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:34.550220 15833 log.cc:824] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:34.552019 15834 tablet_bootstrap.cc:492] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Bootstrap starting.
I20260619 06:40:34.552971 15833 tablet_bootstrap.cc:492] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d: No bootstrap required, opened a new log
I20260619 06:40:34.553231 15833 ts_tablet_manager.cc:1397] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 06:40:34.553273 15836 tablet_bootstrap.cc:492] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Bootstrap starting.
I20260619 06:40:34.555227 15834 tablet_bootstrap.cc:654] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:34.556170 15834 log.cc:824] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:34.556555 15836 tablet_bootstrap.cc:654] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:34.557739 15836 log.cc:824] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:34.558921 15834 tablet_bootstrap.cc:492] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff: No bootstrap required, opened a new log
I20260619 06:40:34.559177 15834 ts_tablet_manager.cc:1397] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260619 06:40:34.560487 15836 tablet_bootstrap.cc:492] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: No bootstrap required, opened a new log
I20260619 06:40:34.560770 15836 ts_tablet_manager.cc:1397] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 06:40:34.563094 15833 raft_consensus.cc:348] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.563531 15833 raft_consensus.cc:374] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:34.563627 15833 raft_consensus.cc:729] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0240319ecc14b0cb857acff473aa64d, State: Initialized, Role: FOLLOWER
I20260619 06:40:34.564262 15833 consensus_queue.cc:260] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.566022 15833 ts_tablet_manager.cc:1428] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260619 06:40:34.568531 15834 raft_consensus.cc:348] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.569036 15834 raft_consensus.cc:374] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:34.569129 15834 raft_consensus.cc:729] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5158f15c63bc4ff8a6fbcb1fc35274ff, State: Initialized, Role: FOLLOWER
I20260619 06:40:34.569285 15836 raft_consensus.cc:348] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.569680 15836 raft_consensus.cc:374] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:34.569772 15836 raft_consensus.cc:729] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d0ec6ca3214ddfa1ba7734d6bba4e3, State: Initialized, Role: FOLLOWER
I20260619 06:40:34.569753 15834 consensus_queue.cc:260] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.570331 15836 consensus_queue.cc:260] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.571537 15834 ts_tablet_manager.cc:1428] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260619 06:40:34.573017 15836 ts_tablet_manager.cc:1428] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260619 06:40:34.585853 15839 raft_consensus.cc:482] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:40:34.586113 15839 raft_consensus.cc:504] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.587702 15839 leader_election.cc:288] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797), 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193:36523)
I20260619 06:40:34.593997 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.588201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:45423 (local address 127.7.146.193:36523)
0619 06:40:34.588386 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:34.588393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:34.589088 (+ 695us) server_negotiation.cc:407] Connection header received
0619 06:40:34.589246 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:34.589250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:34.589305 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:34.589373 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:34.590059 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.590739 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.592390 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.593162 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.593312 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.593385 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:34.593654 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 06:40:34.593772 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:40:34.594027 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.588162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:47403 (local address 127.7.146.194:42797)
0619 06:40:34.588360 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:34.588366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:34.589045 (+ 679us) server_negotiation.cc:407] Connection header received
0619 06:40:34.589219 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:34.589222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:34.589281 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:34.589362 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:34.589979 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.590673 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.592396 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.593175 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.593334 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.593406 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:34.593689 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 06:40:34.593810 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:40:34.594282 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.588130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:42797 (local address 127.7.146.195:47403)
0619 06:40:34.588934 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:34.588955 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:34.589099 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:34.589376 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:34.589380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:34.589426 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:34.589845 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.589855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.590785 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.590791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.592252 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.592262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.593423 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.593431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.593553 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.593575 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:40:34.593654 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:40:34.593774 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":32,"spinlock_wait_cycles":1280,"thread_start_us":471,"threads_started":1}
I20260619 06:40:34.594450 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.588184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:36523 (local address 127.7.146.195:45423)
0619 06:40:34.588987 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:34.589000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:34.589120 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:34.589392 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:34.589396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:34.589436 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:34.589899 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.589908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.590863 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.590866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.592251 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.592262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.593384 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.593390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.593516 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.593539 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:40:34.593619 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:40:34.593741 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":5,"spinlock_wait_cycles":5376,"thread_start_us":439,"threads_started":1}
I20260619 06:40:34.595256 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db10f126476f4189a8bc70d44597de23" candidate_uuid: "b0240319ecc14b0cb857acff473aa64d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" is_pre_election: true
I20260619 06:40:34.595309 15480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db10f126476f4189a8bc70d44597de23" candidate_uuid: "b0240319ecc14b0cb857acff473aa64d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" is_pre_election: true
I20260619 06:40:34.595702 15619 raft_consensus.cc:2454] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0240319ecc14b0cb857acff473aa64d in term 0.
I20260619 06:40:34.595757 15480 raft_consensus.cc:2454] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0240319ecc14b0cb857acff473aa64d in term 0.
I20260619 06:40:34.596318 15691 leader_election.cc:302] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5158f15c63bc4ff8a6fbcb1fc35274ff, b0240319ecc14b0cb857acff473aa64d; no voters:
I20260619 06:40:34.596697 15839 raft_consensus.cc:2781] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:40:34.596823 15839 raft_consensus.cc:482] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:40:34.596921 15839 raft_consensus.cc:3037] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:34.599124 15839 raft_consensus.cc:504] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.599699 15839 leader_election.cc:288] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [CANDIDATE]: Term 1 election: Requested vote from peers 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797), 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193:36523)
I20260619 06:40:34.600098 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db10f126476f4189a8bc70d44597de23" candidate_uuid: "b0240319ecc14b0cb857acff473aa64d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff"
I20260619 06:40:34.600144 15480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db10f126476f4189a8bc70d44597de23" candidate_uuid: "b0240319ecc14b0cb857acff473aa64d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3"
I20260619 06:40:34.600281 15619 raft_consensus.cc:3037] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:34.600308 15480 raft_consensus.cc:3037] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:34.602391 15619 raft_consensus.cc:2454] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0240319ecc14b0cb857acff473aa64d in term 1.
I20260619 06:40:34.602402 15480 raft_consensus.cc:2454] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0240319ecc14b0cb857acff473aa64d in term 1.
I20260619 06:40:34.602756 15692 leader_election.cc:302] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d0ec6ca3214ddfa1ba7734d6bba4e3, b0240319ecc14b0cb857acff473aa64d; no voters:
I20260619 06:40:34.603017 15839 raft_consensus.cc:2781] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:34.603425 15839 raft_consensus.cc:686] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 1 LEADER]: Becoming Leader. State: Replica: b0240319ecc14b0cb857acff473aa64d, State: Running, Role: LEADER
I20260619 06:40:34.603892 15839 consensus_queue.cc:237] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.608078 15323 catalog_manager.cc:5495] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0240319ecc14b0cb857acff473aa64d (127.7.146.195). New cstate: current_term: 1 leader_uuid: "b0240319ecc14b0cb857acff473aa64d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } health_report { overall_health: HEALTHY } } }
I20260619 06:40:34.632704 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.627868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.7.146.195:35333)
0619 06:40:34.628049 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:34.628053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:34.628070 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:40:34.628234 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:34.628237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:34.628315 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:34.628429 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:40:34.628733 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.629396 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.630652 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.631041 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.631129 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.631562 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:40:34.632102 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:40:34.632203 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:34.632360 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 06:40:34.632423 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:40:34.640444 15619 raft_consensus.cc:1261] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Refusing update from remote peer b0240319ecc14b0cb857acff473aa64d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:34.640471 15480 raft_consensus.cc:1261] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Refusing update from remote peer b0240319ecc14b0cb857acff473aa64d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:34.641440 15844 consensus_queue.cc:1035] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:34.641775 15839 consensus_queue.cc:1035] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:34.656234 15847 mvcc.cc:205] Tried to move back new op lower bound from 7298462657078226944 to 7298462656941264896. Current Snapshot: MvccSnapshot[applied={T|T < 7298462657078226944}]
I20260619 06:40:34.675634 15323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:34.675961 15323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:34.680012 15323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49: Sending DeleteTablet for 3 replicas of tablet db10f126476f4189a8bc70d44597de23
I20260619 06:40:34.681192 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet db10f126476f4189a8bc70d44597de23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:34 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:50674
I20260619 06:40:34.681138 15599 tablet_service.cc:1514] Processing DeleteTablet for tablet db10f126476f4189a8bc70d44597de23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:34 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:52848
I20260619 06:40:34.682047 15857 tablet_replica.cc:330] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d: stopping tablet replica
I20260619 06:40:34.682116 15460 tablet_service.cc:1514] Processing DeleteTablet for tablet db10f126476f4189a8bc70d44597de23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:34 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:44242
I20260619 06:40:34.682593 15857 raft_consensus.cc:2229] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:40:34.682435 15323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:40:34.683192 15323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:40:34.683594 15857 raft_consensus.cc:2258] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:34.684710 15598 tablet_service.cc:1514] Processing DeleteTablet for tablet db10f126476f4189a8bc70d44597de23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:34 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:52848
I20260619 06:40:34.688380 15857 ts_tablet_manager.cc:1905] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:34.691121 15598 tablet_service.cc:1467] Processing CreateTablet for tablet 360d9d4689944d85a27a51dc7fc76e66 (DEFAULT_TABLE table=test-table [id=04c44094d12e4d20abdd1ee681edbc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:34.691545 15459 tablet_service.cc:1467] Processing CreateTablet for tablet 360d9d4689944d85a27a51dc7fc76e66 (DEFAULT_TABLE table=test-table [id=04c44094d12e4d20abdd1ee681edbc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:34.691766 15598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360d9d4689944d85a27a51dc7fc76e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:34.692042 15459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360d9d4689944d85a27a51dc7fc76e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:34.695562 15857 ts_tablet_manager.cc:1918] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:34.696174 15859 tablet_replica.cc:330] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: stopping tablet replica
I20260619 06:40:34.696661 15859 raft_consensus.cc:2229] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:34.697224 15859 raft_consensus.cc:2258] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:34.697631 15857 log.cc:1196] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/db10f126476f4189a8bc70d44597de23
I20260619 06:40:34.695684 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 360d9d4689944d85a27a51dc7fc76e66 (DEFAULT_TABLE table=test-table [id=04c44094d12e4d20abdd1ee681edbc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:34.698293 15857 ts_tablet_manager.cc:1939] T db10f126476f4189a8bc70d44597de23 P b0240319ecc14b0cb857acff473aa64d: Deleting consensus metadata
I20260619 06:40:34.698565 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360d9d4689944d85a27a51dc7fc76e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:34.699364 15834 tablet_bootstrap.cc:492] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Bootstrap starting.
I20260619 06:40:34.699663 15859 ts_tablet_manager.cc:1905] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:34.700934 15834 tablet_bootstrap.cc:654] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:34.702277 15836 tablet_bootstrap.cc:492] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Bootstrap starting.
I20260619 06:40:34.703872 15836 tablet_bootstrap.cc:654] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:34.704849 15833 tablet_bootstrap.cc:492] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d: Bootstrap starting.
I20260619 06:40:34.706208 15308 catalog_manager.cc:4841] TS b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333): tablet db10f126476f4189a8bc70d44597de23 (table test-table [id=514d1a5c14a84dd4aa920398dc3041ad]) successfully deleted
I20260619 06:40:34.706306 15833 tablet_bootstrap.cc:654] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:34.710391 15859 ts_tablet_manager.cc:1918] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:34.710572 15859 log.cc:1196] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/db10f126476f4189a8bc70d44597de23
I20260619 06:40:34.711086 15859 ts_tablet_manager.cc:1939] T db10f126476f4189a8bc70d44597de23 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting consensus metadata
I20260619 06:40:34.712852 15310 catalog_manager.cc:4841] TS 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193:36523): tablet db10f126476f4189a8bc70d44597de23 (table test-table [id=514d1a5c14a84dd4aa920398dc3041ad]) successfully deleted
I20260619 06:40:34.714416 15858 tablet_replica.cc:330] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff: stopping tablet replica
I20260619 06:40:34.714702 15834 tablet_bootstrap.cc:492] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff: No bootstrap required, opened a new log
I20260619 06:40:34.714860 15834 ts_tablet_manager.cc:1397] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20260619 06:40:34.714859 15858 raft_consensus.cc:2229] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:34.715718 15834 raft_consensus.cc:348] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.715946 15834 raft_consensus.cc:374] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:34.716056 15834 raft_consensus.cc:729] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5158f15c63bc4ff8a6fbcb1fc35274ff, State: Initialized, Role: FOLLOWER
I20260619 06:40:34.716467 15858 raft_consensus.cc:2258] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:34.716372 15834 consensus_queue.cc:260] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.717842 15834 ts_tablet_manager.cc:1428] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:40:34.718905 15833 tablet_bootstrap.cc:492] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d: No bootstrap required, opened a new log
I20260619 06:40:34.719030 15836 tablet_bootstrap.cc:492] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: No bootstrap required, opened a new log
I20260619 06:40:34.719161 15833 ts_tablet_manager.cc:1397] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260619 06:40:34.719542 15836 ts_tablet_manager.cc:1397] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20260619 06:40:34.720078 15833 raft_consensus.cc:348] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.720356 15833 raft_consensus.cc:374] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:34.720309 15836 raft_consensus.cc:348] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.720552 15836 raft_consensus.cc:374] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:34.720609 15833 raft_consensus.cc:729] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0240319ecc14b0cb857acff473aa64d, State: Initialized, Role: FOLLOWER
I20260619 06:40:34.720682 15836 raft_consensus.cc:729] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d0ec6ca3214ddfa1ba7734d6bba4e3, State: Initialized, Role: FOLLOWER
I20260619 06:40:34.720925 15836 consensus_queue.cc:260] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.721017 15833 consensus_queue.cc:260] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.721441 15836 ts_tablet_manager.cc:1428] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:40:34.722777 15833 ts_tablet_manager.cc:1428] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 06:40:34.723575 15858 ts_tablet_manager.cc:1905] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:34.729313 15858 ts_tablet_manager.cc:1918] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:34.729455 15858 log.cc:1196] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/db10f126476f4189a8bc70d44597de23
I20260619 06:40:34.729849 15858 ts_tablet_manager.cc:1939] T db10f126476f4189a8bc70d44597de23 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting consensus metadata
I20260619 06:40:34.731338 15309 catalog_manager.cc:4841] TS 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797): tablet db10f126476f4189a8bc70d44597de23 (table test-table [id=514d1a5c14a84dd4aa920398dc3041ad]) successfully deleted
W20260619 06:40:34.731851 15309 catalog_manager.cc:4808] TS 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797): delete failed for tablet db10f126476f4189a8bc70d44597de23 because the tablet was not found. No further retry: Not found: Tablet not found: db10f126476f4189a8bc70d44597de23
W20260619 06:40:34.945648 15531 tablet.cc:2369] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:40:34.946869 15809 tablet.cc:2369] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:40:34.947691 15841 raft_consensus.cc:482] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:40:34.947901 15841 raft_consensus.cc:504] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
W20260619 06:40:34.949093 15670 tablet.cc:2369] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:40:34.949702 15841 leader_election.cc:288] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797), b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333)
I20260619 06:40:34.966378 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.950472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:33727 (local address 127.7.146.195:35333)
0619 06:40:34.950619 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:34.950625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:34.958668 (+ 8043us) server_negotiation.cc:407] Connection header received
0619 06:40:34.958896 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:34.958901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:34.959003 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:34.959065 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:34.959893 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.961087 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.963833 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.965120 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.965246 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.965327 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:34.965912 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 06:40:34.966074 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 06:40:34.966460 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.950324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:48579 (local address 127.7.146.194:42797)
0619 06:40:34.950513 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:34.950520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:34.958466 (+ 7946us) server_negotiation.cc:407] Connection header received
0619 06:40:34.958677 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:34.958683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:34.958769 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:34.958892 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:34.959860 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.961041 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.963828 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.965154 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.965307 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.965394 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:34.966006 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 06:40:34.966144 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:40:34.967095 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.950133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:35333 (local address 127.7.146.193:33727)
0619 06:40:34.958581 (+ 8448us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:34.958597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:34.958745 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:34.959280 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:34.959284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:34.959325 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:34.959738 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.959748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.961253 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.961258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.963662 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.963676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.965520 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.965527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.965721 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.965757 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:40:34.966579 (+ 822us) client_negotiation.cc:240] Negotiation successful
0619 06:40:34.966721 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8329,"mutex_wait_us":62,"thread_start_us":7858,"threads_started":1}
I20260619 06:40:34.967689 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:34.949985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:42797 (local address 127.7.146.193:48579)
0619 06:40:34.958303 (+ 8318us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:34.958326 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:34.958530 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:34.959165 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:34.959170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:34.959199 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:34.959625 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.959639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.961228 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.961234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.963638 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:34.963654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:34.965604 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:34.965611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:34.965813 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:34.965847 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:40:34.967002 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0619 06:40:34.967342 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8162,"mutex_wait_us":79,"thread_start_us":8341,"threads_started":1}
I20260619 06:40:34.967878 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360d9d4689944d85a27a51dc7fc76e66" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0240319ecc14b0cb857acff473aa64d" is_pre_election: true
I20260619 06:40:34.968299 15758 raft_consensus.cc:2454] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 0.
I20260619 06:40:34.968384 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360d9d4689944d85a27a51dc7fc76e66" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" is_pre_election: true
I20260619 06:40:34.968652 15619 raft_consensus.cc:2454] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 0.
I20260619 06:40:34.968837 15411 leader_election.cc:302] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d0ec6ca3214ddfa1ba7734d6bba4e3, b0240319ecc14b0cb857acff473aa64d; no voters:
I20260619 06:40:34.969247 15841 raft_consensus.cc:2781] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:40:34.969353 15841 raft_consensus.cc:482] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:40:34.969435 15841 raft_consensus.cc:3037] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:34.971444 15841 raft_consensus.cc:504] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.972051 15841 leader_election.cc:288] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 election: Requested vote from peers 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797), b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333)
I20260619 06:40:34.972477 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360d9d4689944d85a27a51dc7fc76e66" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff"
I20260619 06:40:34.972558 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360d9d4689944d85a27a51dc7fc76e66" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0240319ecc14b0cb857acff473aa64d"
I20260619 06:40:34.972720 15619 raft_consensus.cc:3037] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:34.972781 15758 raft_consensus.cc:3037] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:34.975157 15758 raft_consensus.cc:2454] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 1.
I20260619 06:40:34.975559 15411 leader_election.cc:302] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d0ec6ca3214ddfa1ba7734d6bba4e3, b0240319ecc14b0cb857acff473aa64d; no voters:
I20260619 06:40:34.975672 15619 raft_consensus.cc:2454] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 1.
I20260619 06:40:34.975822 15841 raft_consensus.cc:2781] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:34.976274 15841 raft_consensus.cc:686] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 LEADER]: Becoming Leader. State: Replica: 29d0ec6ca3214ddfa1ba7734d6bba4e3, State: Running, Role: LEADER
I20260619 06:40:34.976840 15841 consensus_queue.cc:237] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:34.980520 15321 catalog_manager.cc:5495] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193). New cstate: current_term: 1 leader_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } health_report { overall_health: UNKNOWN } } }
Jun 19 06:40:35 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851231, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.254@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:35.021205 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.007275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34441 (local address 127.0.0.1:59762)
0619 06:40:35.008183 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:35.008201 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:35.008359 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:35.008665 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:35.008669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:35.009478 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:35.009735 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:35.009744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.010797 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.010801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:35.011738 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:35.011749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.012420 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.012427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:35.012632 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.013388 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:35.013409 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:35.015853 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:35.018811 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:35.018824 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:35.018842 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:35.019218 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:35.019548 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:35.019552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:35.019556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:35.019683 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:35.020149 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:35.020157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:35.020316 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 06:40:35.020455 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 06:40:35.020771 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":557,"threads_started":1}
I20260619 06:40:35.021302 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.007217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.7.146.254:34441)
0619 06:40:35.007954 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:35.007961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:35.008301 (+ 340us) server_negotiation.cc:407] Connection header received
0619 06:40:35.008472 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:35.008476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:35.008528 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:35.008639 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:35.009887 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.010679 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.011862 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.012253 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.012354 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.016089 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:35.016094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:35.016097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:35.016124 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:35.018677 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:35.019321 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:35.019334 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:35.019336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:35.019451 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:35.019773 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:35.019778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:35.019781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:35.020032 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:35.020155 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:35.020556 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 06:40:35.020922 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":552,"thread_start_us":334,"threads_started":1}
I20260619 06:40:35.026381 15321 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:59762:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:40:35.027428 15321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:40:35.034583 15459 tablet_service.cc:1467] Processing CreateTablet for tablet 1570fb2e4fb140429f0701240f626946 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96b41ca2fd15405d8064d07b06317233]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:40:35.034564 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 1570fb2e4fb140429f0701240f626946 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96b41ca2fd15405d8064d07b06317233]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:40:35.035115 15459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1570fb2e4fb140429f0701240f626946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:35.035146 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1570fb2e4fb140429f0701240f626946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:35.034924 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 1570fb2e4fb140429f0701240f626946 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96b41ca2fd15405d8064d07b06317233]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:40:35.035523 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1570fb2e4fb140429f0701240f626946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:35.042147 15836 tablet_bootstrap.cc:492] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Bootstrap starting.
I20260619 06:40:35.042147 15834 tablet_bootstrap.cc:492] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Bootstrap starting.
I20260619 06:40:35.043370 15833 tablet_bootstrap.cc:492] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d: Bootstrap starting.
I20260619 06:40:35.043743 15836 tablet_bootstrap.cc:654] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:35.043744 15834 tablet_bootstrap.cc:654] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:35.045099 15833 tablet_bootstrap.cc:654] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:35.054668 15834 tablet_bootstrap.cc:492] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff: No bootstrap required, opened a new log
I20260619 06:40:35.054864 15834 ts_tablet_manager.cc:1397] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260619 06:40:35.055583 15833 tablet_bootstrap.cc:492] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d: No bootstrap required, opened a new log
I20260619 06:40:35.055609 15834 raft_consensus.cc:348] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.055868 15834 raft_consensus.cc:374] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:35.055956 15834 raft_consensus.cc:729] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5158f15c63bc4ff8a6fbcb1fc35274ff, State: Initialized, Role: FOLLOWER
I20260619 06:40:35.056080 15836 tablet_bootstrap.cc:492] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: No bootstrap required, opened a new log
I20260619 06:40:35.056069 15833 ts_tablet_manager.cc:1397] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260619 06:40:35.056262 15836 ts_tablet_manager.cc:1397] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260619 06:40:35.056248 15834 consensus_queue.cc:260] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.056704 15840 tablet_replica.cc:439] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } } }
I20260619 06:40:35.057067 15840 tablet_replica.cc:442] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:40:35.056975 15836 raft_consensus.cc:348] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.057194 15836 raft_consensus.cc:374] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:35.057269 15836 raft_consensus.cc:729] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d0ec6ca3214ddfa1ba7734d6bba4e3, State: Initialized, Role: FOLLOWER
I20260619 06:40:35.057278 15833 raft_consensus.cc:348] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.057495 15836 consensus_queue.cc:260] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.057617 15833 raft_consensus.cc:374] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:35.057835 15833 raft_consensus.cc:729] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0240319ecc14b0cb857acff473aa64d, State: Initialized, Role: FOLLOWER
I20260619 06:40:35.057888 15841 tablet_replica.cc:439] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } } }
I20260619 06:40:35.058148 15836 ts_tablet_manager.cc:1428] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 06:40:35.058264 15833 consensus_queue.cc:260] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.058167 15841 tablet_replica.cc:442] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:40:35.058811 15854 tablet_replica.cc:439] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } } }
I20260619 06:40:35.059185 15854 tablet_replica.cc:442] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:40:35.059242 15834 ts_tablet_manager.cc:1428] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 06:40:35.062637 15833 ts_tablet_manager.cc:1428] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260619 06:40:35.140975 15841 raft_consensus.cc:482] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:40:35.141186 15841 raft_consensus.cc:504] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.141770 15841 leader_election.cc:288] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797), b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333)
I20260619 06:40:35.142405 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1570fb2e4fb140429f0701240f626946" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" is_pre_election: true
I20260619 06:40:35.142416 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1570fb2e4fb140429f0701240f626946" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0240319ecc14b0cb857acff473aa64d" is_pre_election: true
I20260619 06:40:35.142640 15619 raft_consensus.cc:2454] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 0.
I20260619 06:40:35.142685 15758 raft_consensus.cc:2454] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 0.
I20260619 06:40:35.143090 15412 leader_election.cc:302] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29d0ec6ca3214ddfa1ba7734d6bba4e3, 5158f15c63bc4ff8a6fbcb1fc35274ff, b0240319ecc14b0cb857acff473aa64d; no voters:
I20260619 06:40:35.143381 15841 raft_consensus.cc:2781] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:40:35.143477 15841 raft_consensus.cc:482] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:40:35.143582 15841 raft_consensus.cc:3037] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:35.145614 15841 raft_consensus.cc:504] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.146173 15841 leader_election.cc:288] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 election: Requested vote from peers 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797), b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333)
I20260619 06:40:35.146656 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1570fb2e4fb140429f0701240f626946" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0240319ecc14b0cb857acff473aa64d"
I20260619 06:40:35.146647 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1570fb2e4fb140429f0701240f626946" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff"
I20260619 06:40:35.146870 15758 raft_consensus.cc:3037] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:35.146870 15619 raft_consensus.cc:3037] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:35.149686 15758 raft_consensus.cc:2454] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 1.
I20260619 06:40:35.149688 15619 raft_consensus.cc:2454] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 1.
I20260619 06:40:35.150056 15412 leader_election.cc:302] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d0ec6ca3214ddfa1ba7734d6bba4e3, 5158f15c63bc4ff8a6fbcb1fc35274ff; no voters:
I20260619 06:40:35.150360 15841 raft_consensus.cc:2781] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:35.150521 15841 raft_consensus.cc:686] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 LEADER]: Becoming Leader. State: Replica: 29d0ec6ca3214ddfa1ba7734d6bba4e3, State: Running, Role: LEADER
I20260619 06:40:35.150770 15841 consensus_queue.cc:237] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.152262 15865 tablet_replica.cc:439] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: TxnStatusTablet state changed. Reason: New leader 29d0ec6ca3214ddfa1ba7734d6bba4e3. Latest consensus state: current_term: 1 leader_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } } }
I20260619 06:40:35.152537 15865 tablet_replica.cc:442] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:40:35.153141 15878 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:40:35.153316 15878 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:40:35.153586 15322 catalog_manager.cc:5495] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193). New cstate: current_term: 1 leader_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } health_report { overall_health: UNKNOWN } } }
I20260619 06:40:35.188267 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.181437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:36523 (local address 127.0.0.1:44258)
0619 06:40:35.181684 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:35.181704 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:35.181857 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:35.182672 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:35.182678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:35.182690 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:40:35.182909 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:35.182918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.184456 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.184460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:35.185509 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:35.185522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.186474 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.186478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:35.186636 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.186693 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:40:35.187763 (+ 1070us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:40:35.187782 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:40:35.187847 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 06:40:35.188037 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 06:40:35.188306 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.181583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.7.146.193:36523)
0619 06:40:35.182225 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:35.182231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:35.182251 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:40:35.182348 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:35.182353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:35.182455 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:35.182657 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:40:35.183060 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.184315 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.185642 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.186318 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.186449 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.186876 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:40:35.187651 (+ 775us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:40:35.187798 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:35.187873 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 06:40:35.187954 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":328,"threads_started":1}
I20260619 06:40:35.192598 15619 raft_consensus.cc:1261] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Refusing update from remote peer 29d0ec6ca3214ddfa1ba7734d6bba4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:35.192898 15758 raft_consensus.cc:1261] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Refusing update from remote peer 29d0ec6ca3214ddfa1ba7734d6bba4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:35.193305 15865 consensus_queue.cc:1035] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:35.193631 15841 consensus_queue.cc:1035] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:35.200973 15854 tablet_replica.cc:439] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d: TxnStatusTablet state changed. Reason: New leader 29d0ec6ca3214ddfa1ba7734d6bba4e3. Latest consensus state: current_term: 1 leader_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } } }
I20260619 06:40:35.201231 15854 tablet_replica.cc:442] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:40:35.208078 15840 tablet_replica.cc:439] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff: TxnStatusTablet state changed. Reason: New leader 29d0ec6ca3214ddfa1ba7734d6bba4e3. Latest consensus state: current_term: 1 leader_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } } }
I20260619 06:40:35.208374 15840 tablet_replica.cc:442] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:40:35.209056 15882 mvcc.cc:205] Tried to move back new op lower bound from 7298462659342000128 to 7298462659179835392. Current Snapshot: MvccSnapshot[applied={T|T < 7298462659342000128}]
I20260619 06:40:35.213650 15865 tablet_replica.cc:439] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } } }
I20260619 06:40:35.213960 15865 tablet_replica.cc:442] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:40:35.216670 15865 tablet_replica.cc:439] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } } }
I20260619 06:40:35.217029 15865 tablet_replica.cc:442] T 1570fb2e4fb140429f0701240f626946 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:40:35.222797 15854 tablet_replica.cc:439] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } } }
I20260619 06:40:35.223037 15854 tablet_replica.cc:442] T 1570fb2e4fb140429f0701240f626946 P b0240319ecc14b0cb857acff473aa64d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:40:35.225023 15840 tablet_replica.cc:439] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } } }
I20260619 06:40:35.225275 15840 tablet_replica.cc:442] T 1570fb2e4fb140429f0701240f626946 P 5158f15c63bc4ff8a6fbcb1fc35274ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:40:35.234419 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.228603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.7.146.193:36523)
0619 06:40:35.228738 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:35.228742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:35.229166 (+ 424us) server_negotiation.cc:407] Connection header received
0619 06:40:35.229299 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:35.229305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:35.229384 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:35.229478 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:40:35.229835 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.231009 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.232391 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.233091 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.233218 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.233598 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:40:35.233730 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:40:35.233822 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:35.234037 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 06:40:35.234117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 06:40:35.250802 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.244110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:43927 (local address 127.7.146.193:36523)
0619 06:40:35.244306 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:35.244315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:35.244907 (+ 592us) server_negotiation.cc:407] Connection header received
0619 06:40:35.245094 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:35.245099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:35.245188 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:35.245291 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:40:35.245789 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.246988 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.248719 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.249395 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.249544 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.250027 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:40:35.250129 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:40:35.250206 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:35.250320 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 06:40:35.250435 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":43}
I20260619 06:40:35.250910 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.244211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:36523 (local address 127.7.146.193:43927)
0619 06:40:35.244816 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:35.244834 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:35.244960 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:35.245297 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:35.245300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:35.245308 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:40:35.245646 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:35.245654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.247134 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.247138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:35.248551 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:35.248596 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.249599 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.249606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:35.249796 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.249871 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:40:35.250215 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:40:35.250224 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:40:35.250294 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:40:35.250482 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":17,"thread_start_us":292,"threads_started":1}
I20260619 06:40:35.260377 15758 raft_consensus.cc:1261] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Refusing update from remote peer 29d0ec6ca3214ddfa1ba7734d6bba4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:35.260429 15618 raft_consensus.cc:1261] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Refusing update from remote peer 29d0ec6ca3214ddfa1ba7734d6bba4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:35.260881 15865 consensus_queue.cc:1035] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:35.261116 15841 consensus_queue.cc:1035] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:35.273347 15883 mvcc.cc:205] Tried to move back new op lower bound from 7298462659621941248 to 7298462658468753408. Current Snapshot: MvccSnapshot[applied={T|T < 7298462659621941248}]
I20260619 06:40:35.320801 15323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:35.320995 15323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:35.323918 15323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49: Sending DeleteTablet for 3 replicas of tablet 360d9d4689944d85a27a51dc7fc76e66
I20260619 06:40:35.324776 15460 tablet_service.cc:1514] Processing DeleteTablet for tablet 360d9d4689944d85a27a51dc7fc76e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:35 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:44242
I20260619 06:40:35.324939 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet 360d9d4689944d85a27a51dc7fc76e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:35 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:50674
I20260619 06:40:35.325569 15599 tablet_service.cc:1514] Processing DeleteTablet for tablet 360d9d4689944d85a27a51dc7fc76e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:35 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:52848
I20260619 06:40:35.341383 15900 tablet_replica.cc:330] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d: stopping tablet replica
I20260619 06:40:35.342185 15900 raft_consensus.cc:2229] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:35.342744 15900 raft_consensus.cc:2258] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:35.344362 15900 ts_tablet_manager.cc:1905] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:35.347272 15901 tablet_replica.cc:330] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff: stopping tablet replica
I20260619 06:40:35.347455 15901 raft_consensus.cc:2229] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:35.347707 15901 raft_consensus.cc:2258] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:35.348269 15899 tablet_replica.cc:330] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: stopping tablet replica
I20260619 06:40:35.348470 15899 raft_consensus.cc:2229] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:40:35.349126 15899 raft_consensus.cc:2258] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:35.350004 15901 ts_tablet_manager.cc:1905] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:35.353092 15899 ts_tablet_manager.cc:1905] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:35.354149 15900 ts_tablet_manager.cc:1918] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:40:35.354326 15900 log.cc:1196] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/360d9d4689944d85a27a51dc7fc76e66
I20260619 06:40:35.354847 15900 ts_tablet_manager.cc:1939] T 360d9d4689944d85a27a51dc7fc76e66 P b0240319ecc14b0cb857acff473aa64d: Deleting consensus metadata
I20260619 06:40:35.355988 15308 catalog_manager.cc:4841] TS b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333): tablet 360d9d4689944d85a27a51dc7fc76e66 (table test-table [id=04c44094d12e4d20abdd1ee681edbc1e]) successfully deleted
I20260619 06:40:35.359957 15899 ts_tablet_manager.cc:1918] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:40:35.360095 15899 log.cc:1196] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/360d9d4689944d85a27a51dc7fc76e66
I20260619 06:40:35.360522 15899 ts_tablet_manager.cc:1939] T 360d9d4689944d85a27a51dc7fc76e66 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting consensus metadata
I20260619 06:40:35.361621 15310 catalog_manager.cc:4841] TS 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193:36523): tablet 360d9d4689944d85a27a51dc7fc76e66 (table test-table [id=04c44094d12e4d20abdd1ee681edbc1e]) successfully deleted
I20260619 06:40:35.361944 15901 ts_tablet_manager.cc:1918] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:40:35.362076 15901 log.cc:1196] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/360d9d4689944d85a27a51dc7fc76e66
I20260619 06:40:35.362502 15901 ts_tablet_manager.cc:1939] T 360d9d4689944d85a27a51dc7fc76e66 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting consensus metadata
I20260619 06:40:35.363579 15309 catalog_manager.cc:4841] TS 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797): tablet 360d9d4689944d85a27a51dc7fc76e66 (table test-table [id=04c44094d12e4d20abdd1ee681edbc1e]) successfully deleted
Jun 19 06:40:35 dist-test-slave-qfk4 krb5kdc[15278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:40:35 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:35 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:40:35.388389 15872 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:35.376893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.7.146.254:34441)
0619 06:40:35.377081 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:35.377086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:35.377503 (+ 417us) server_negotiation.cc:407] Connection header received
0619 06:40:35.377699 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:35.377704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:35.377786 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:35.377882 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:35.379236 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.380029 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.381322 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.381753 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.381867 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.388140 (+ 6273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59772: BlockingRecv error: recv got EOF from 127.0.0.1:59772 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 06:40:35 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM
I20260619 06:40:35.405723 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.390922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.7.146.254:34441)
0619 06:40:35.391098 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:35.391103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:35.391384 (+ 281us) server_negotiation.cc:407] Connection header received
0619 06:40:35.391535 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:35.391541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:35.391619 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:35.391721 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:35.392896 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.393977 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.395223 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.395861 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.395985 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.399750 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:35.399755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:35.399760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:35.399799 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:35.403085 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:35.403769 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:35.403775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:35.403779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:35.403927 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:35.404228 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:35.404233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:35.404237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:35.404593 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:35.404831 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:35.405165 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 06:40:35.405395 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:40:35.409300 15323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:40:35.409968 15323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:40:35.416617 15460 tablet_service.cc:1467] Processing CreateTablet for tablet 1f06850a93f84b5da3e3ccf17d9b2722 (DEFAULT_TABLE table=test-table [id=de81763b860a4c80b95486ff49e25088]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:35.417017 15460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f06850a93f84b5da3e3ccf17d9b2722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:35.421048 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 1f06850a93f84b5da3e3ccf17d9b2722 (DEFAULT_TABLE table=test-table [id=de81763b860a4c80b95486ff49e25088]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:35.421530 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f06850a93f84b5da3e3ccf17d9b2722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:35.421952 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 1f06850a93f84b5da3e3ccf17d9b2722 (DEFAULT_TABLE table=test-table [id=de81763b860a4c80b95486ff49e25088]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:35.422327 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f06850a93f84b5da3e3ccf17d9b2722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:35.426239 15836 tablet_bootstrap.cc:492] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Bootstrap starting.
I20260619 06:40:35.427546 15834 tablet_bootstrap.cc:492] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Bootstrap starting.
I20260619 06:40:35.427951 15836 tablet_bootstrap.cc:654] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:35.428037 15833 tablet_bootstrap.cc:492] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d: Bootstrap starting.
I20260619 06:40:35.429044 15834 tablet_bootstrap.cc:654] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:35.429617 15833 tablet_bootstrap.cc:654] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:35.439960 15834 tablet_bootstrap.cc:492] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff: No bootstrap required, opened a new log
I20260619 06:40:35.440127 15834 ts_tablet_manager.cc:1397] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260619 06:40:35.441075 15834 raft_consensus.cc:348] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.441332 15834 raft_consensus.cc:374] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:35.441431 15834 raft_consensus.cc:729] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5158f15c63bc4ff8a6fbcb1fc35274ff, State: Initialized, Role: FOLLOWER
I20260619 06:40:35.441644 15834 consensus_queue.cc:260] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.442265 15834 ts_tablet_manager.cc:1428] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:40:35.448426 15836 tablet_bootstrap.cc:492] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: No bootstrap required, opened a new log
I20260619 06:40:35.448623 15836 ts_tablet_manager.cc:1397] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.009s
I20260619 06:40:35.449486 15836 raft_consensus.cc:348] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.450083 15836 raft_consensus.cc:374] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:35.450191 15836 raft_consensus.cc:729] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d0ec6ca3214ddfa1ba7734d6bba4e3, State: Initialized, Role: FOLLOWER
I20260619 06:40:35.450479 15836 consensus_queue.cc:260] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.454654 15836 ts_tablet_manager.cc:1428] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260619 06:40:35.458436 15833 tablet_bootstrap.cc:492] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d: No bootstrap required, opened a new log
I20260619 06:40:35.458595 15833 ts_tablet_manager.cc:1397] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.007s
I20260619 06:40:35.459355 15833 raft_consensus.cc:348] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.459627 15833 raft_consensus.cc:374] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:35.459713 15833 raft_consensus.cc:729] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0240319ecc14b0cb857acff473aa64d, State: Initialized, Role: FOLLOWER
I20260619 06:40:35.459920 15833 consensus_queue.cc:260] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.462729 15833 ts_tablet_manager.cc:1428] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 06:40:35.627341 15840 raft_consensus.cc:482] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:40:35.627554 15840 raft_consensus.cc:504] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.629068 15840 leader_election.cc:288] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193:36523), b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333)
I20260619 06:40:35.636480 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.629735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:44513 (local address 127.7.146.195:35333)
0619 06:40:35.630698 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:35.630703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:35.630720 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:40:35.630788 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:35.630792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:35.630854 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:35.631002 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:35.631451 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.632529 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.634513 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.635478 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.635630 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.635710 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:35.636029 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 06:40:35.636164 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":701,"threads_started":1}
I20260619 06:40:35.636538 15917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.629295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:35333 (local address 127.7.146.194:44513)
0619 06:40:35.630061 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:35.630086 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:35.630230 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:35.630995 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:35.631000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:35.631026 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:35.631263 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:35.631276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.632669 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.632674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:35.634347 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:35.634359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.635636 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.635642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:35.635818 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.635853 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:40:35.635998 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 06:40:35.636139 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":90,"thread_start_us":554,"threads_started":1}
I20260619 06:40:35.637439 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f06850a93f84b5da3e3ccf17d9b2722" candidate_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0240319ecc14b0cb857acff473aa64d" is_pre_election: true
I20260619 06:40:35.637693 15758 raft_consensus.cc:2454] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5158f15c63bc4ff8a6fbcb1fc35274ff in term 0.
I20260619 06:40:35.637753 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.629334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:36523 (local address 127.7.146.194:48319)
0619 06:40:35.630959 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:35.630977 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:35.631164 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:35.631529 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:35.631533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:35.631565 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:40:35.631828 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:35.631836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.632916 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.632922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:35.635155 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:35.635170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.636976 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.636981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:35.637150 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.637181 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:40:35.637269 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:40:35.637386 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1507,"mutex_wait_us":3,"thread_start_us":1299,"threads_started":1}
I20260619 06:40:35.637741 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.629436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:48319 (local address 127.7.146.193:36523)
0619 06:40:35.629652 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:35.629659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:35.631058 (+ 1399us) server_negotiation.cc:407] Connection header received
0619 06:40:35.631329 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:35.631334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:35.631414 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:35.631498 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:40:35.631961 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.632782 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.635324 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.636588 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.636734 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.636799 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:35.637318 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 06:40:35.637472 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 06:40:35.638319 15480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f06850a93f84b5da3e3ccf17d9b2722" candidate_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" is_pre_election: true
I20260619 06:40:35.638593 15480 raft_consensus.cc:2454] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5158f15c63bc4ff8a6fbcb1fc35274ff in term 0.
I20260619 06:40:35.638573 15551 leader_election.cc:302] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5158f15c63bc4ff8a6fbcb1fc35274ff, b0240319ecc14b0cb857acff473aa64d; no voters:
I20260619 06:40:35.639010 15840 raft_consensus.cc:2781] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:40:35.639147 15840 raft_consensus.cc:482] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:40:35.639266 15840 raft_consensus.cc:3037] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:35.642428 15840 raft_consensus.cc:504] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.643132 15840 leader_election.cc:288] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [CANDIDATE]: Term 1 election: Requested vote from peers 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193:36523), b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333)
I20260619 06:40:35.643646 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f06850a93f84b5da3e3ccf17d9b2722" candidate_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0240319ecc14b0cb857acff473aa64d"
I20260619 06:40:35.643716 15480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f06850a93f84b5da3e3ccf17d9b2722" candidate_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3"
I20260619 06:40:35.643863 15758 raft_consensus.cc:3037] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:35.643918 15480 raft_consensus.cc:3037] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:35.646291 15480 raft_consensus.cc:2454] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5158f15c63bc4ff8a6fbcb1fc35274ff in term 1.
I20260619 06:40:35.646713 15553 leader_election.cc:302] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d0ec6ca3214ddfa1ba7734d6bba4e3, 5158f15c63bc4ff8a6fbcb1fc35274ff; no voters:
I20260619 06:40:35.646899 15758 raft_consensus.cc:2454] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5158f15c63bc4ff8a6fbcb1fc35274ff in term 1.
I20260619 06:40:35.647028 15840 raft_consensus.cc:2781] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:35.647588 15840 raft_consensus.cc:686] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 LEADER]: Becoming Leader. State: Replica: 5158f15c63bc4ff8a6fbcb1fc35274ff, State: Running, Role: LEADER
I20260619 06:40:35.647969 15840 consensus_queue.cc:237] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.651485 15323 catalog_manager.cc:5495] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194). New cstate: current_term: 1 leader_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } health_report { overall_health: UNKNOWN } } }
I20260619 06:40:35.676240 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.669398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.7.146.194:42797)
0619 06:40:35.670013 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:35.670019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:35.670043 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:40:35.670147 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:35.670152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:35.670256 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:35.670443 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:40:35.670791 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.671960 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.673506 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.674152 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.674280 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.674723 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:40:35.675535 (+ 812us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:40:35.675677 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:35.675795 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 06:40:35.675881 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":323,"threads_started":1}
I20260619 06:40:35.680162 15758 raft_consensus.cc:1261] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Refusing update from remote peer 5158f15c63bc4ff8a6fbcb1fc35274ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:35.680174 15480 raft_consensus.cc:1261] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Refusing update from remote peer 5158f15c63bc4ff8a6fbcb1fc35274ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:35.680845 15840 consensus_queue.cc:1035] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:35.681172 15920 consensus_queue.cc:1035] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:35.707250 15322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:35.707527 15322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:35.710827 15322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49: Sending DeleteTablet for 3 replicas of tablet 1f06850a93f84b5da3e3ccf17d9b2722
I20260619 06:40:35.711644 15460 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f06850a93f84b5da3e3ccf17d9b2722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:35 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:44242
I20260619 06:40:35.711949 15899 tablet_replica.cc:330] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: stopping tablet replica
I20260619 06:40:35.711980 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f06850a93f84b5da3e3ccf17d9b2722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:35 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:50674
I20260619 06:40:35.712150 15899 raft_consensus.cc:2229] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:35.712235 15900 tablet_replica.cc:330] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d: stopping tablet replica
I20260619 06:40:35.712373 15899 raft_consensus.cc:2258] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:35.712430 15900 raft_consensus.cc:2229] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:35.712682 15900 raft_consensus.cc:2258] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:35.713706 15900 ts_tablet_manager.cc:1905] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:35.713745 15599 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f06850a93f84b5da3e3ccf17d9b2722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:35 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:52848
I20260619 06:40:35.714303 15901 tablet_replica.cc:330] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff: stopping tablet replica
I20260619 06:40:35.714512 15901 raft_consensus.cc:2229] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:40:35.714993 15901 raft_consensus.cc:2258] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:35.716095 15901 ts_tablet_manager.cc:1905] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:35.715886 15322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:40:35.716795 15322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:40:35.720146 15899 ts_tablet_manager.cc:1905] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:35.722783 15901 ts_tablet_manager.cc:1918] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:35.722954 15901 log.cc:1196] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/1f06850a93f84b5da3e3ccf17d9b2722
I20260619 06:40:35.723377 15901 ts_tablet_manager.cc:1939] T 1f06850a93f84b5da3e3ccf17d9b2722 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting consensus metadata
I20260619 06:40:35.724733 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 55f307519d404be8b4ecfc7551890a03 (DEFAULT_TABLE table=test-table [id=c9d5c448bd28406d81614d9f21ed148a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:35.725050 15309 catalog_manager.cc:4841] TS 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797): tablet 1f06850a93f84b5da3e3ccf17d9b2722 (table test-table [id=de81763b860a4c80b95486ff49e25088]) successfully deleted
I20260619 06:40:35.725155 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f307519d404be8b4ecfc7551890a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:35.725310 15738 tablet_service.cc:1467] Processing CreateTablet for tablet 55f307519d404be8b4ecfc7551890a03 (DEFAULT_TABLE table=test-table [id=c9d5c448bd28406d81614d9f21ed148a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:35.725729 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f307519d404be8b4ecfc7551890a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:35.727227 15900 ts_tablet_manager.cc:1918] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:35.727370 15900 log.cc:1196] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/1f06850a93f84b5da3e3ccf17d9b2722
I20260619 06:40:35.727450 15460 tablet_service.cc:1467] Processing CreateTablet for tablet 55f307519d404be8b4ecfc7551890a03 (DEFAULT_TABLE table=test-table [id=c9d5c448bd28406d81614d9f21ed148a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:40:35.727777 15900 ts_tablet_manager.cc:1939] T 1f06850a93f84b5da3e3ccf17d9b2722 P b0240319ecc14b0cb857acff473aa64d: Deleting consensus metadata
I20260619 06:40:35.727898 15460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f307519d404be8b4ecfc7551890a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:35.729115 15308 catalog_manager.cc:4841] TS b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333): tablet 1f06850a93f84b5da3e3ccf17d9b2722 (table test-table [id=de81763b860a4c80b95486ff49e25088]) successfully deleted
I20260619 06:40:35.730886 15834 tablet_bootstrap.cc:492] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Bootstrap starting.
I20260619 06:40:35.732796 15834 tablet_bootstrap.cc:654] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:35.745437 15836 tablet_bootstrap.cc:492] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Bootstrap starting.
I20260619 06:40:35.747219 15836 tablet_bootstrap.cc:654] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:35.747455 15833 tablet_bootstrap.cc:492] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d: Bootstrap starting.
I20260619 06:40:35.749341 15833 tablet_bootstrap.cc:654] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:35.752302 15834 tablet_bootstrap.cc:492] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff: No bootstrap required, opened a new log
I20260619 06:40:35.752499 15834 ts_tablet_manager.cc:1397] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.008s
I20260619 06:40:35.755371 15834 raft_consensus.cc:348] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.755657 15834 raft_consensus.cc:374] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:35.755736 15834 raft_consensus.cc:729] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5158f15c63bc4ff8a6fbcb1fc35274ff, State: Initialized, Role: FOLLOWER
I20260619 06:40:35.756415 15899 ts_tablet_manager.cc:1918] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:40:35.756769 15899 log.cc:1196] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/1f06850a93f84b5da3e3ccf17d9b2722
I20260619 06:40:35.756671 15834 consensus_queue.cc:260] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.757893 15834 ts_tablet_manager.cc:1428] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 06:40:35.761473 15836 tablet_bootstrap.cc:492] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: No bootstrap required, opened a new log
I20260619 06:40:35.761670 15836 ts_tablet_manager.cc:1397] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20260619 06:40:35.762482 15836 raft_consensus.cc:348] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.762737 15836 raft_consensus.cc:374] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:35.762815 15836 raft_consensus.cc:729] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d0ec6ca3214ddfa1ba7734d6bba4e3, State: Initialized, Role: FOLLOWER
I20260619 06:40:35.763075 15836 consensus_queue.cc:260] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.764307 15836 ts_tablet_manager.cc:1428] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 06:40:35.765008 15899 ts_tablet_manager.cc:1939] T 1f06850a93f84b5da3e3ccf17d9b2722 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting consensus metadata
I20260619 06:40:35.766366 15310 catalog_manager.cc:4841] TS 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193:36523): tablet 1f06850a93f84b5da3e3ccf17d9b2722 (table test-table [id=de81763b860a4c80b95486ff49e25088]) successfully deleted
I20260619 06:40:35.769651 15833 tablet_bootstrap.cc:492] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d: No bootstrap required, opened a new log
I20260619 06:40:35.769824 15833 ts_tablet_manager.cc:1397] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.000s
I20260619 06:40:35.770570 15833 raft_consensus.cc:348] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.770833 15833 raft_consensus.cc:374] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:35.770911 15833 raft_consensus.cc:729] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0240319ecc14b0cb857acff473aa64d, State: Initialized, Role: FOLLOWER
I20260619 06:40:35.771179 15833 consensus_queue.cc:260] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.771798 15833 ts_tablet_manager.cc:1428] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:40:35.772176 15865 raft_consensus.cc:482] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:40:35.772342 15865 raft_consensus.cc:504] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.773021 15865 leader_election.cc:288] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797), b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333)
I20260619 06:40:35.773471 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f307519d404be8b4ecfc7551890a03" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" is_pre_election: true
I20260619 06:40:35.773734 15619 raft_consensus.cc:2454] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 0.
I20260619 06:40:35.774240 15412 leader_election.cc:302] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d0ec6ca3214ddfa1ba7734d6bba4e3, 5158f15c63bc4ff8a6fbcb1fc35274ff; no voters:
I20260619 06:40:35.774343 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f307519d404be8b4ecfc7551890a03" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0240319ecc14b0cb857acff473aa64d" is_pre_election: true
I20260619 06:40:35.774560 15758 raft_consensus.cc:2454] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 0.
I20260619 06:40:35.774820 15865 raft_consensus.cc:2781] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:40:35.774937 15865 raft_consensus.cc:482] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:40:35.775022 15865 raft_consensus.cc:3037] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:35.779254 15865 raft_consensus.cc:504] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.780372 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f307519d404be8b4ecfc7551890a03" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff"
I20260619 06:40:35.780701 15619 raft_consensus.cc:3037] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:35.781687 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f307519d404be8b4ecfc7551890a03" candidate_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0240319ecc14b0cb857acff473aa64d"
I20260619 06:40:35.781865 15758 raft_consensus.cc:3037] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:35.783954 15619 raft_consensus.cc:2454] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 1.
I20260619 06:40:35.784787 15758 raft_consensus.cc:2454] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d0ec6ca3214ddfa1ba7734d6bba4e3 in term 1.
I20260619 06:40:35.784976 15865 leader_election.cc:288] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 election: Requested vote from peers 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797), b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333)
I20260619 06:40:35.785137 15412 leader_election.cc:302] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d0ec6ca3214ddfa1ba7734d6bba4e3, 5158f15c63bc4ff8a6fbcb1fc35274ff; no voters:
I20260619 06:40:35.785475 15865 raft_consensus.cc:2781] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:35.785655 15865 raft_consensus.cc:686] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 LEADER]: Becoming Leader. State: Replica: 29d0ec6ca3214ddfa1ba7734d6bba4e3, State: Running, Role: LEADER
I20260619 06:40:35.785982 15865 consensus_queue.cc:237] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } }
I20260619 06:40:35.788894 15322 catalog_manager.cc:5495] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193). New cstate: current_term: 1 leader_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d0ec6ca3214ddfa1ba7734d6bba4e3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 36523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 } health_report { overall_health: UNKNOWN } } }
I20260619 06:40:35.823462 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.816799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.7.146.193:36523)
0619 06:40:35.816955 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:35.816960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:35.816976 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:40:35.817034 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:35.817039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:35.817134 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:35.817216 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:40:35.817785 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.818904 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.820859 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.821566 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.821679 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.822280 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:40:35.822431 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:40:35.822506 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:35.823051 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 06:40:35.823137 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:40:35.839679 15619 raft_consensus.cc:1261] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Refusing update from remote peer 29d0ec6ca3214ddfa1ba7734d6bba4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:35.840356 15865 consensus_queue.cc:1035] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5158f15c63bc4ff8a6fbcb1fc35274ff" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:35.841029 15758 raft_consensus.cc:1261] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Refusing update from remote peer 29d0ec6ca3214ddfa1ba7734d6bba4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:40:35.841681 15865 consensus_queue.cc:1035] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0240319ecc14b0cb857acff473aa64d" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:40:35.931088 15323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:35.931380 15323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:40:35.935102 15323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d2effef6fef431c9369a36274396f49: Sending DeleteTablet for 3 replicas of tablet 55f307519d404be8b4ecfc7551890a03
I20260619 06:40:35.936012 15459 tablet_service.cc:1514] Processing DeleteTablet for tablet 55f307519d404be8b4ecfc7551890a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:35 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:44242
I20260619 06:40:35.936209 15599 tablet_service.cc:1514] Processing DeleteTablet for tablet 55f307519d404be8b4ecfc7551890a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:35 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:52848
I20260619 06:40:35.936548 15899 tablet_replica.cc:330] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: stopping tablet replica
I20260619 06:40:35.936770 15899 raft_consensus.cc:2229] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:40:35.936821 15901 tablet_replica.cc:330] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff: stopping tablet replica
I20260619 06:40:35.937012 15901 raft_consensus.cc:2229] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:35.937176 15899 raft_consensus.cc:2258] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:35.937242 15901 raft_consensus.cc:2258] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:35.938082 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet 55f307519d404be8b4ecfc7551890a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:40:35 UTC) from {username='oryx', principal='oryx/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:50674
I20260619 06:40:35.938349 15900 tablet_replica.cc:330] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d: stopping tablet replica
I20260619 06:40:35.938459 15901 ts_tablet_manager.cc:1905] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:35.938513 15900 raft_consensus.cc:2229] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:40:35.938689 15900 raft_consensus.cc:2258] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:40:35.939798 15900 ts_tablet_manager.cc:1905] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:35.945824 15901 ts_tablet_manager.cc:1918] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:40:35.946043 15901 log.cc:1196] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/55f307519d404be8b4ecfc7551890a03
I20260619 06:40:35.946547 15901 ts_tablet_manager.cc:1939] T 55f307519d404be8b4ecfc7551890a03 P 5158f15c63bc4ff8a6fbcb1fc35274ff: Deleting consensus metadata
I20260619 06:40:35.947662 15899 ts_tablet_manager.cc:1905] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:40:35.947911 15309 catalog_manager.cc:4841] TS 5158f15c63bc4ff8a6fbcb1fc35274ff (127.7.146.194:42797): tablet 55f307519d404be8b4ecfc7551890a03 (table test-table [id=c9d5c448bd28406d81614d9f21ed148a]) successfully deleted
I20260619 06:40:35.952270 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 15394
I20260619 06:40:35.956278 15900 ts_tablet_manager.cc:1918] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:40:35.961560 15899 ts_tablet_manager.cc:1918] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:40:35.961724 15899 log.cc:1196] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/55f307519d404be8b4ecfc7551890a03
I20260619 06:40:35.961884 15900 log.cc:1196] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/55f307519d404be8b4ecfc7551890a03
I20260619 06:40:35.962191 15899 ts_tablet_manager.cc:1939] T 55f307519d404be8b4ecfc7551890a03 P 29d0ec6ca3214ddfa1ba7734d6bba4e3: Deleting consensus metadata
I20260619 06:40:35.962479 15900 ts_tablet_manager.cc:1939] T 55f307519d404be8b4ecfc7551890a03 P b0240319ecc14b0cb857acff473aa64d: Deleting consensus metadata
I20260619 06:40:35.963384 15310 catalog_manager.cc:4841] TS 29d0ec6ca3214ddfa1ba7734d6bba4e3 (127.7.146.193:36523): tablet 55f307519d404be8b4ecfc7551890a03 (table test-table [id=c9d5c448bd28406d81614d9f21ed148a]) successfully deleted
I20260619 06:40:35.963914 15308 catalog_manager.cc:4841] TS b0240319ecc14b0cb857acff473aa64d (127.7.146.195:35333): tablet 55f307519d404be8b4ecfc7551890a03 (table test-table [id=c9d5c448bd28406d81614d9f21ed148a]) successfully deleted
Jun 19 06:40:35 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.146.193@KRBTEST.COM
I20260619 06:40:35.990509 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:35.955499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.7.146.193:36523)
0619 06:40:35.955733 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:35.955739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:35.971039 (+ 15300us) server_negotiation.cc:407] Connection header received
0619 06:40:35.971538 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:35.971545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:35.971631 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:35.971701 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:35.973815 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.974938 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.976715 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:35.977366 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:35.977472 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:35.982881 (+ 5409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:35.982900 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:35.982918 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:35.982981 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:35.987024 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:35.987902 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:35.987913 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:35.987917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:35.988069 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:35.988526 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:35.988531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:35.988534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:35.989028 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:35.989337 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:35.989907 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 06:40:35.990152 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:40:35.991959 15525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:36.143002 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 15394
I20260619 06:40:36.172976 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 15534
Jun 19 06:40:36 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.146.194@KRBTEST.COM
I20260619 06:40:36.188690 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:36.173830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.7.146.194:42797)
0619 06:40:36.174053 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:36.174060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:36.174496 (+ 436us) server_negotiation.cc:407] Connection header received
0619 06:40:36.174742 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:36.174748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:36.174834 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:36.174922 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:36.176151 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:36.176981 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:36.178178 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:36.178606 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:36.178708 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:36.183074 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:36.183088 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:36.183112 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:36.183157 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:36.186061 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:36.186676 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:36.186684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:36.186686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:36.186816 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:36.187154 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:36.187157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:36.187160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:36.187522 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:36.187805 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:36.188115 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 06:40:36.188330 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:40:36.189330 15664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:36.319450 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 15534
I20260619 06:40:36.337272 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 15673
Jun 19 06:40:36 dist-test-slave-qfk4 krb5kdc[15278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.146.195@KRBTEST.COM
I20260619 06:40:36.351876 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:36.337992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50684 (local address 127.7.146.195:35333)
0619 06:40:36.338488 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:36.338492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:36.338508 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:40:36.338563 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:36.338567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:36.338626 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:36.338718 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:36.340021 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:36.340835 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:36.342039 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:36.342434 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:36.342535 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:36.346938 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:36.346985 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:36.347001 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:36.347053 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:36.349641 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:36.350250 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:36.350257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:36.350259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:36.350382 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:36.350650 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:36.350653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:36.350655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:36.350995 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:36.351203 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:36.351423 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 06:40:36.351579 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":288,"threads_started":1}
I20260619 06:40:36.352519 15803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:36.472648 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 15673
I20260619 06:40:36.489168 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 15291
I20260619 06:40:36.490309 15363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:36.604147 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 15291
2026-06-19T06:40:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6437 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:40:36 dist-test-slave-qfk4 krb5kdc[15946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:40:36 dist-test-slave-qfk4 krb5kdc[15946](info): set up 2 sockets
Jun 19 06:40:36 dist-test-slave-qfk4 krb5kdc[15946](info): commencing operation
krb5kdc: starting...
W20260619 06:40:37.613361 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:40:37 dist-test-slave-qfk4 krb5kdc[15946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:40:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:40:37.819248 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34625
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:34625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:38.130020 15959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:38.130306 15959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:38.130420 15959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:38.137652 15959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:38.137760 15959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:38.137838 15959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:38.137912 15959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:38.137974 15959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:38.147776 15959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:34625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:38.149084 15959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:38.150357 15959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:38.161528 15964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:38.161576 15965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:38.161541 15967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:38.162503 15959 server_base.cc:1034] running on GCE node
I20260619 06:40:38.163285 15959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:38.164747 15959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:38.165954 15959 hybrid_clock.cc:648] HybridClock initialized: now 1781851238165896 us; error 52 us; skew 500 ppm
I20260619 06:40:38.166350 15959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T06:40:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1634 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:40:38 dist-test-slave-qfk4 krb5kdc[15973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:40:38 dist-test-slave-qfk4 krb5kdc[15973](info): set up 2 sockets
Jun 19 06:40:38 dist-test-slave-qfk4 krb5kdc[15973](info): commencing operation
krb5kdc: starting...
W20260619 06:40:39.261412 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:40:39 dist-test-slave-qfk4 krb5kdc[15973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:40:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:40:39.471850 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:39047
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:39047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:39.789320 15986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:39.789647 15986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:39.789790 15986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:39.797120 15986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:39.797254 15986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:39.797343 15986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:39.797418 15986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:39.797482 15986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:39.807442 15986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:39047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:39047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:39.808794 15986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:39.810104 15986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:39.821195 15992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:39.821244 15991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:39.821470 15994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:39.822059 15986 server_base.cc:1034] running on GCE node
I20260619 06:40:39.822877 15986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:39.824321 15986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:39.825543 15986 hybrid_clock.cc:648] HybridClock initialized: now 1781851239825483 us; error 50 us; skew 500 ppm
I20260619 06:40:39.825891 15986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:39 dist-test-slave-qfk4 krb5kdc[15973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851239, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:39.828960 15986 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:40:39.830482 15986 webserver.cc:460] Webserver started at http://127.7.146.254:38961/ using document root <none> and password file <none>
I20260619 06:40:39.831182 15986 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:39.831295 15986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:39.831583 15986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:39.834672 15986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "ec477f6d8e1248419d2011aa2b8fedf0"
format_stamp: "Formatted at 2026-06-19 06:40:39 on dist-test-slave-qfk4"
server_key: "9cf1d6e9d213823d8b0e29e4e6218d47"
server_key_iv: "2da230c4289e712f3e55c41d02f86a37"
server_key_version: "encryptionkey@0"
I20260619 06:40:39.835507 15986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "ec477f6d8e1248419d2011aa2b8fedf0"
format_stamp: "Formatted at 2026-06-19 06:40:39 on dist-test-slave-qfk4"
server_key: "9cf1d6e9d213823d8b0e29e4e6218d47"
server_key_iv: "2da230c4289e712f3e55c41d02f86a37"
server_key_version: "encryptionkey@0"
I20260619 06:40:39.841151 15986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 06:40:39.845198 16001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:39.846690 15986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:40:39.846884 15986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "ec477f6d8e1248419d2011aa2b8fedf0"
format_stamp: "Formatted at 2026-06-19 06:40:39 on dist-test-slave-qfk4"
server_key: "9cf1d6e9d213823d8b0e29e4e6218d47"
server_key_iv: "2da230c4289e712f3e55c41d02f86a37"
server_key_version: "encryptionkey@0"
I20260619 06:40:39.847085 15986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:39.870731 15986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:39.875008 15986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:39.875262 15986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:39.892115 15986 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:39047
I20260619 06:40:39.892136 16052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:39047 every 8 connection(s)
I20260619 06:40:39.893471 15986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:39.897310 16053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:39.901544 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 15986
I20260619 06:40:39.901736 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:39.902390 7755 external_mini_cluster.cc:1428] Setting key b6dbfcc3f839a817a12403cecc0ba76d
I20260619 06:40:39.908134 16053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0: Bootstrap starting.
I20260619 06:40:39.911195 16053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:39.912166 16053 log.cc:824] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:39.915565 16053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0: No bootstrap required, opened a new log
Jun 19 06:40:39 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:39.923374 16053 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } }
I20260619 06:40:39.923794 16053 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:39.923890 16053 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec477f6d8e1248419d2011aa2b8fedf0, State: Initialized, Role: FOLLOWER
I20260619 06:40:39.924440 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:39.904346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.7.146.254:39047)
0619 06:40:39.905218 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:39.905230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:39.905289 (+ 59us) server_negotiation.cc:407] Connection header received
0619 06:40:39.907032 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:39.907055 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:39.907488 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:39.907865 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:39.909318 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:39.910644 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:39.911880 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:39.912641 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:39.912854 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:39.917286 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:39.917297 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:39.917313 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:39.917353 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:39.920351 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:39.921083 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:39.921093 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:39.921100 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:39.921269 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:39.921642 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:39.921646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:39.921648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:39.922213 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:39.922504 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:39.922930 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 06:40:39.923340 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":435,"threads_started":1}
I20260619 06:40:39.924512 16053 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } }
I20260619 06:40:39.924826 16053 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:39.924970 16053 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:39.925151 16053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:39.927681 16053 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } }
I20260619 06:40:39.928130 16053 leader_election.cc:302] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec477f6d8e1248419d2011aa2b8fedf0; no voters:
I20260619 06:40:39.928723 16053 leader_election.cc:288] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:39.928843 16058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:39.929436 16058 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 1 LEADER]: Becoming Leader. State: Replica: ec477f6d8e1248419d2011aa2b8fedf0, State: Running, Role: LEADER
I20260619 06:40:39.930007 16058 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } }
I20260619 06:40:39.930796 16053 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:39.932899 16060 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec477f6d8e1248419d2011aa2b8fedf0. Latest consensus state: current_term: 1 leader_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } } }
I20260619 06:40:39.932871 16059 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } } }
I20260619 06:40:39.933422 16060 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:39.933432 16059 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:39.935124 16066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:39.940390 16066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:39.948648 16066 catalog_manager.cc:1347] Generated new cluster ID: 1bcc008fc10541e89e47126949e6db05
I20260619 06:40:39.948770 16066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:39.959441 16066 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:39.960868 16066 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:39.977311 16066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0: Generated new TSK 0
I20260619 06:40:39.978163 16066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:40:40.036970 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:39047
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:40.361886 16079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:40.362174 16079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:40.362308 16079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:40.369576 16079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:40.369691 16079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:40.369835 16079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:40:40.379467 16079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:40.380811 16079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:40.382077 16079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:40.394349 16084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:40.394425 16085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:40.394389 16087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:40.395151 16079 server_base.cc:1034] running on GCE node
I20260619 06:40:40.395709 16079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:40.396585 16079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:40.397794 16079 hybrid_clock.cc:648] HybridClock initialized: now 1781851240397724 us; error 66 us; skew 500 ppm
I20260619 06:40:40.398087 16079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:40 dist-test-slave-qfk4 krb5kdc[15973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851240, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:40.401136 16079 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:40:40.402469 16079 webserver.cc:460] Webserver started at http://127.7.146.193:37533/ using document root <none> and password file <none>
I20260619 06:40:40.403107 16079 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:40.403200 16079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:40.403443 16079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:40.406275 16079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "06e1c89602fc4d8daf101a5b0812b387"
format_stamp: "Formatted at 2026-06-19 06:40:40 on dist-test-slave-qfk4"
server_key: "05f7321dbe4c44b4221bd79927e1be77"
server_key_iv: "6029b996dce82945b8255d94d13443d4"
server_key_version: "encryptionkey@0"
I20260619 06:40:40.406965 16079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "06e1c89602fc4d8daf101a5b0812b387"
format_stamp: "Formatted at 2026-06-19 06:40:40 on dist-test-slave-qfk4"
server_key: "05f7321dbe4c44b4221bd79927e1be77"
server_key_iv: "6029b996dce82945b8255d94d13443d4"
server_key_version: "encryptionkey@0"
I20260619 06:40:40.411680 16079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 06:40:40.414969 16094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:40.416235 16079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:40:40.416373 16079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "06e1c89602fc4d8daf101a5b0812b387"
format_stamp: "Formatted at 2026-06-19 06:40:40 on dist-test-slave-qfk4"
server_key: "05f7321dbe4c44b4221bd79927e1be77"
server_key_iv: "6029b996dce82945b8255d94d13443d4"
server_key_version: "encryptionkey@0"
I20260619 06:40:40.416544 16079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:40.439499 16079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:40.444029 16079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:40.444350 16079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:40.445459 16079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:40.446918 16079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:40.447003 16079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:40.447139 16079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:40.447198 16079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:40.479278 16079 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:36959
I20260619 06:40:40.479362 16206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:36959 every 8 connection(s)
I20260619 06:40:40.480688 16079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:40:40.487528 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 16079
I20260619 06:40:40.487706 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:40:40.488509 7755 external_mini_cluster.cc:1428] Setting key 2fdd183794666e9e0831fdb30dcb945d
Jun 19 06:40:40 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851240, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:40.501024 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:40.483149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:38327 (local address 127.7.146.254:39047)
0619 06:40:40.483584 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:40.483591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:40.484546 (+ 955us) server_negotiation.cc:407] Connection header received
0619 06:40:40.486371 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:40.486375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:40.486445 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:40.486542 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:40.488044 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:40.489143 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:40.490930 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:40.491543 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:40.491680 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:40.494990 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:40.494998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:40.495003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:40.495045 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:40.498033 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:40.498897 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:40.498901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:40.498903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:40.499030 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:40.499457 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:40.499459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:40.499461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:40.499704 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:40.499818 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:40.500555 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 06:40:40.500765 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":204,"threads_started":1}
I20260619 06:40:40.502354 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:40.483683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:38327)
0619 06:40:40.484388 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:40.484427 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:40.486103 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:40.486724 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:40.486732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:40.487237 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:40.487875 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:40.487894 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:40.489368 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:40.489373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:40.490789 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:40.490800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:40.491762 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:40.491768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:40.491968 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:40.492654 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:40.492677 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:40.494752 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:40.498173 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:40.498182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:40.498200 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:40.498775 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:40.499124 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:40.499129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:40.499132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:40.499363 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:40.499816 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:40.499824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:40.500201 (+ 377us) client_negotiation.cc:769] Sending connection context
0619 06:40:40.500488 (+ 287us) client_negotiation.cc:240] Negotiation successful
0619 06:40:40.501033 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":261,"threads_started":1}
I20260619 06:40:40.503815 16207 heartbeater.cc:347] Connected to a master server at 127.7.146.254:39047
I20260619 06:40:40.504101 16207 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:40.504803 16207 heartbeater.cc:511] Master 127.7.146.254:39047 requested a full tablet report, sending...
I20260619 06:40:40.506755 16018 ts_manager.cc:194] Registered new tserver with Master: 06e1c89602fc4d8daf101a5b0812b387 (127.7.146.193:36959)
I20260619 06:40:40.508173 16018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:38327
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:40:40.535565 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:39047
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:40.846071 16213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:40.846355 16213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:40.846475 16213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:40.854327 16213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:40.854447 16213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:40.854590 16213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:40:40.864362 16213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:40.865694 16213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:40.867022 16213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:40.879675 16218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:40.879724 16221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:40.879649 16219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:40.880676 16213 server_base.cc:1034] running on GCE node
I20260619 06:40:40.881235 16213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:40.882167 16213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:40.883363 16213 hybrid_clock.cc:648] HybridClock initialized: now 1781851240883302 us; error 55 us; skew 500 ppm
I20260619 06:40:40.883671 16213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:40 dist-test-slave-qfk4 krb5kdc[15973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851240, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:40.886835 16213 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:40:40.888331 16213 webserver.cc:460] Webserver started at http://127.7.146.194:44209/ using document root <none> and password file <none>
I20260619 06:40:40.889058 16213 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:40.889176 16213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:40.889498 16213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:40.892728 16213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "be7f771c5c7049f093657b7d6c9e1983"
format_stamp: "Formatted at 2026-06-19 06:40:40 on dist-test-slave-qfk4"
server_key: "e6750ffb201acbbaebb771a96e7936d7"
server_key_iv: "b1cabf0035acf83e743744f473d58c8e"
server_key_version: "encryptionkey@0"
I20260619 06:40:40.893551 16213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "be7f771c5c7049f093657b7d6c9e1983"
format_stamp: "Formatted at 2026-06-19 06:40:40 on dist-test-slave-qfk4"
server_key: "e6750ffb201acbbaebb771a96e7936d7"
server_key_iv: "b1cabf0035acf83e743744f473d58c8e"
server_key_version: "encryptionkey@0"
I20260619 06:40:40.898967 16213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:40:40.902673 16228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:40.904053 16213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:40:40.904194 16213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "be7f771c5c7049f093657b7d6c9e1983"
format_stamp: "Formatted at 2026-06-19 06:40:40 on dist-test-slave-qfk4"
server_key: "e6750ffb201acbbaebb771a96e7936d7"
server_key_iv: "b1cabf0035acf83e743744f473d58c8e"
server_key_version: "encryptionkey@0"
I20260619 06:40:40.904385 16213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:40.925858 16213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:40.929633 16213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:40.929883 16213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:40.930850 16213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:40.932101 16213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:40.932178 16213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:40.932291 16213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:40.932343 16213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:40.962335 16213 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:44769
I20260619 06:40:40.962402 16340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:44769 every 8 connection(s)
I20260619 06:40:40.963584 16213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:40:40.965592 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 16213
I20260619 06:40:40.965750 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:40:40.966213 7755 external_mini_cluster.cc:1428] Setting key cc5f25d10a30e190c19d5b8344531cfd
Jun 19 06:40:40 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851240, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:40.984222 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:40.966301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:55397 (local address 127.7.146.254:39047)
0619 06:40:40.966496 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:40.966502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:40.967949 (+ 1447us) server_negotiation.cc:407] Connection header received
0619 06:40:40.969839 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:40.969844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:40.969934 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:40.970037 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:40.971655 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:40.972634 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:40.974249 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:40.974661 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:40.974751 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:40.978609 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:40.978615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:40.978618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:40.978650 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:40.981047 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:40.981699 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:40.981703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:40.981706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:40.981822 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:40.982293 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:40.982299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:40.982303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:40.982690 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:40.982863 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:40.983588 (+ 725us) server_negotiation.cc:299] Negotiation successful
0619 06:40:40.983827 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:40:40.985088 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:40.966862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:55397)
0619 06:40:40.967694 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:40.967753 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:40.969556 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:40.970307 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:40.970316 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:40.970857 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:40.971475 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:40.971493 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:40.972775 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:40.972780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:40.974094 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:40.974109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:40.974880 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:40.974887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:40.975183 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:40.975861 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:40.975888 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:40.978390 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:40.981170 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:40.981180 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:40.981197 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:40.981590 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:40.981932 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:40.981936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:40.981940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:40.982148 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:40.982857 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:40.982863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:40.983192 (+ 329us) client_negotiation.cc:769] Sending connection context
0619 06:40:40.983444 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 06:40:40.983900 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":316,"threads_started":1}
I20260619 06:40:40.986481 16341 heartbeater.cc:347] Connected to a master server at 127.7.146.254:39047
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
I20260619 06:40:40.986832 16341 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:40.987591 16341 heartbeater.cc:511] Master 127.7.146.254:39047 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
I20260619 06:40:40.989096 16018 ts_manager.cc:194] Registered new tserver with Master: be7f771c5c7049f093657b7d6c9e1983 (127.7.146.194:44769)
I20260619 06:40:40.990129 16018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:55397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:40:41.013720 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:39047
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:41.334108 16346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:41.334456 16346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:41.334599 16346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:41.342061 16346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:41.342187 16346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:41.342337 16346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:40:41.352442 16346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:41.353823 16346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:41.355420 16346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:41.367889 16351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:41.368479 16352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:41.369314 16354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:41.369727 16346 server_base.cc:1034] running on GCE node
I20260619 06:40:41.370349 16346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:41.371279 16346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:41.372483 16346 hybrid_clock.cc:648] HybridClock initialized: now 1781851241372435 us; error 41 us; skew 500 ppm
I20260619 06:40:41.372804 16346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:41 dist-test-slave-qfk4 krb5kdc[15973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851241, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:41.375742 16346 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:40:41.377187 16346 webserver.cc:460] Webserver started at http://127.7.146.195:42497/ using document root <none> and password file <none>
I20260619 06:40:41.377866 16346 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:41.377985 16346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:41.378257 16346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:41.381093 16346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "5990656ac2db46dd872511764d395436"
format_stamp: "Formatted at 2026-06-19 06:40:41 on dist-test-slave-qfk4"
server_key: "0d71b5a0e25f51b56b7ce2ffaefddd60"
server_key_iv: "60ee416859a6ec349c5302b395bce21a"
server_key_version: "encryptionkey@0"
I20260619 06:40:41.381810 16346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "5990656ac2db46dd872511764d395436"
format_stamp: "Formatted at 2026-06-19 06:40:41 on dist-test-slave-qfk4"
server_key: "0d71b5a0e25f51b56b7ce2ffaefddd60"
server_key_iv: "60ee416859a6ec349c5302b395bce21a"
server_key_version: "encryptionkey@0"
I20260619 06:40:41.386742 16346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:40:41.390168 16361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:41.391594 16346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:40:41.391739 16346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "5990656ac2db46dd872511764d395436"
format_stamp: "Formatted at 2026-06-19 06:40:41 on dist-test-slave-qfk4"
server_key: "0d71b5a0e25f51b56b7ce2ffaefddd60"
server_key_iv: "60ee416859a6ec349c5302b395bce21a"
server_key_version: "encryptionkey@0"
I20260619 06:40:41.391954 16346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:41.410897 16346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:41.414978 16346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:41.415231 16346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:41.416193 16346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:41.417539 16346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:41.417614 16346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:41.417729 16346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:41.417783 16346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:41.447362 16346 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:40467
I20260619 06:40:41.447429 16473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:40467 every 8 connection(s)
I20260619 06:40:41.449121 16346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:40:41.453872 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 16346
I20260619 06:40:41.454066 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:40:41.454583 7755 external_mini_cluster.cc:1428] Setting key 275b9f8ac8757b9f4156c8d584d7f74a
Jun 19 06:40:41 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851241, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:40:41.470687 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:41.451775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:43083 (local address 127.7.146.254:39047)
0619 06:40:41.451971 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:41.451978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:41.453404 (+ 1426us) server_negotiation.cc:407] Connection header received
0619 06:40:41.455765 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:41.455770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:41.455882 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:41.455995 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:41.457868 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:41.458660 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:41.460170 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:41.460868 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:41.460976 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:41.464218 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:41.464223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:41.464226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:41.464262 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:41.467127 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:41.467907 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:41.467911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:41.467913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:41.468054 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:41.468641 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:41.468645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:41.468648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:41.469067 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:41.469197 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:41.470110 (+ 913us) server_negotiation.cc:299] Negotiation successful
0619 06:40:41.470351 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:40:41.471675 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:41.452456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:43083)
0619 06:40:41.453232 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:41.453274 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:41.455457 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:41.456285 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:41.456297 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:41.456995 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:41.457674 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:41.457701 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:41.458784 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:41.458788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:41.460028 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:41.460041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:41.461052 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:41.461057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:41.461248 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:41.461845 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:41.461872 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:41.463983 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:40:41.467288 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:41.467296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:41.467311 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:41.467770 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:41.468168 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:40:41.468172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:40:41.468175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:40:41.468405 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:40:41.469205 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:40:41.469214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:40:41.469636 (+ 422us) client_negotiation.cc:769] Sending connection context
0619 06:40:41.469965 (+ 329us) client_negotiation.cc:240] Negotiation successful
0619 06:40:41.470564 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":271,"threads_started":1}
I20260619 06:40:41.473009 16474 heartbeater.cc:347] Connected to a master server at 127.7.146.254:39047
I20260619 06:40:41.473273 16474 heartbeater.cc:464] Registering TS with master...
I20260619 06:40:41.473942 16474 heartbeater.cc:511] Master 127.7.146.254:39047 requested a full tablet report, sending...
I20260619 06:40:41.475298 16018 ts_manager.cc:194] Registered new tserver with Master: 5990656ac2db46dd872511764d395436 (127.7.146.195:40467)
I20260619 06:40:41.476125 16018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:43083
I20260619 06:40:41.482839 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.193@KRBTEST.COM" created.
I20260619 06:40:41.511543 16207 heartbeater.cc:503] Master 127.7.146.254:39047 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab.
Entry for principal oryx/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab.
WARNING: no policy specified for oryx/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab.
Entry for principal oryx/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab.
WARNING: no policy specified for oryx/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab.
Entry for principal oryx/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab.
I20260619 06:40:41.611076 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 16079
Jun 19 06:40:41 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:40:41.627017 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:41.612197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.7.146.193:36959)
0619 06:40:41.612830 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:41.612836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:41.612865 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:40:41.612935 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:41.612942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:41.613150 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:41.613307 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:41.614736 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:41.615866 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:41.617185 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:41.617753 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:41.617852 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:41.621833 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:41.621846 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:41.621866 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:41.621905 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:41.624470 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:41.625172 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:41.625179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:41.625181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:41.625304 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:41.625726 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:41.625730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:41.625732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:41.626065 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:41.626284 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:41.626551 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 06:40:41.626733 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":307,"threads_started":1}
I20260619 06:40:41.628232 16202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:41.711272 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 16079
I20260619 06:40:41.724817 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 16213
Jun 19 06:40:41 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:40:41.741519 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:41.725739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.7.146.194:44769)
0619 06:40:41.726264 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:41.726271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:41.726302 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:40:41.726362 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:41.726369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:41.726580 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:41.726705 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:41.727982 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:41.729382 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:41.730700 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:41.731327 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:41.731438 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:41.735770 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:41.735782 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:41.735797 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:41.735842 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:41.738652 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:41.739342 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:41.739349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:41.739352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:41.739499 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:41.739877 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:41.739881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:41.739883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:41.740237 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:41.740552 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:41.740896 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 06:40:41.741156 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":264,"threads_started":1}
I20260619 06:40:41.742739 16336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:41.823976 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 16213
I20260619 06:40:41.836256 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 16346
Jun 19 06:40:41 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:40:41.851768 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:41.837144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.7.146.195:40467)
0619 06:40:41.837703 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:41.837710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:41.837739 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:40:41.837798 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:41.837804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:41.838025 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:41.838151 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:41.839358 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:41.840368 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:41.841742 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:41.842291 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:41.842404 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:41.846535 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:41.846546 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:41.846560 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:41.846593 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:41.849238 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:41.849863 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:41.849870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:41.849872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:41.850017 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:41.850392 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:41.850396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:41.850399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:41.850765 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:41.851020 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:41.851232 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 06:40:41.851489 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":270,"threads_started":1}
I20260619 06:40:41.853093 16469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:41.935107 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 16346
I20260619 06:40:41.947304 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 15986
I20260619 06:40:41.948472 16048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:42.049242 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 15986
I20260619 06:40:42.065423 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:39047
--webserver_interface=127.7.146.254
--webserver_port=38961
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:39047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:40:42.379524 16491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:42.379830 16491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:42.379966 16491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:42.387133 16491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:42.387252 16491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:42.387331 16491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:42.387404 16491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:42.387465 16491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:42.397310 16491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:39047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:39047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=38961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:42.398589 16491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:42.399869 16491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:42.410759 16496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:42.410722 16497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:42.410799 16499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:42.411633 16491 server_base.cc:1034] running on GCE node
I20260619 06:40:42.412340 16491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:42.413674 16491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:42.414880 16491 hybrid_clock.cc:648] HybridClock initialized: now 1781851242414838 us; error 35 us; skew 500 ppm
I20260619 06:40:42.415226 16491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:42 dist-test-slave-qfk4 krb5kdc[15973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851242, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:42.418196 16491 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:40:42.419565 16491 webserver.cc:460] Webserver started at http://127.7.146.254:38961/ using document root <none> and password file <none>
I20260619 06:40:42.420203 16491 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:42.420322 16491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:42.425616 16491 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:40:42.428464 16506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:42.429898 16491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:42.430058 16491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "ec477f6d8e1248419d2011aa2b8fedf0"
format_stamp: "Formatted at 2026-06-19 06:40:39 on dist-test-slave-qfk4"
server_key: "9cf1d6e9d213823d8b0e29e4e6218d47"
server_key_iv: "2da230c4289e712f3e55c41d02f86a37"
server_key_version: "encryptionkey@0"
I20260619 06:40:42.430990 16491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:42.449895 16491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:42.454445 16491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:42.454700 16491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:42.470885 16491 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:39047
I20260619 06:40:42.470865 16557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:39047 every 8 connection(s)
I20260619 06:40:42.472638 16491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:42.477909 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 16491
I20260619 06:40:42.478753 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:36959
--local_ip_for_outbound_sockets=127.7.146.193
--tserver_master_addrs=127.7.146.254:39047
--webserver_port=37533
--webserver_interface=127.7.146.193
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.146.193
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:40:42.479877 16558 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:40:42.483222 16558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0: Bootstrap starting.
I20260619 06:40:42.501607 16558 log.cc:824] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:42.509999 16558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:40:42.510468 16558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0: Bootstrap complete.
I20260619 06:40:42.519028 16558 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } }
I20260619 06:40:42.519830 16558 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec477f6d8e1248419d2011aa2b8fedf0, State: Initialized, Role: FOLLOWER
I20260619 06:40:42.520388 16558 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } }
I20260619 06:40:42.520655 16558 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:42.520767 16558 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:42.520938 16558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:40:42.524610 16558 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } }
I20260619 06:40:42.525117 16558 leader_election.cc:302] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec477f6d8e1248419d2011aa2b8fedf0; no voters:
I20260619 06:40:42.525683 16558 leader_election.cc:288] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:40:42.525890 16562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:40:42.526719 16562 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [term 2 LEADER]: Becoming Leader. State: Replica: ec477f6d8e1248419d2011aa2b8fedf0, State: Running, Role: LEADER
I20260619 06:40:42.527309 16562 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } }
I20260619 06:40:42.527566 16558 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:42.529592 16564 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec477f6d8e1248419d2011aa2b8fedf0. Latest consensus state: current_term: 2 leader_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } } }
I20260619 06:40:42.529950 16564 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:42.530400 16563 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec477f6d8e1248419d2011aa2b8fedf0" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39047 } } }
I20260619 06:40:42.530680 16563 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:42.538125 16568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:42.544747 16568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:42.546355 16568 catalog_manager.cc:1259] Loaded cluster ID: 1bcc008fc10541e89e47126949e6db05
I20260619 06:40:42.546525 16568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:42.550674 16568 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:42.552472 16568 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec477f6d8e1248419d2011aa2b8fedf0: Loaded TSK: 0
I20260619 06:40:42.554209 16568 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:40:42.826143 16560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:42.826450 16560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:42.826581 16560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:42.834205 16560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:42.834342 16560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:42.834506 16560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:40:42.844903 16560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.193.keytab
--principal=oryx/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:36959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=37533
--tserver_master_addrs=127.7.146.254:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:42.846247 16560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:42.847605 16560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:42.862175 16586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:42.862569 16585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:42.863154 16588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:42.863497 16560 server_base.cc:1034] running on GCE node
I20260619 06:40:42.864166 16560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:42.865342 16560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:42.866586 16560 hybrid_clock.cc:648] HybridClock initialized: now 1781851242866525 us; error 64 us; skew 500 ppm
I20260619 06:40:42.866940 16560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:42 dist-test-slave-qfk4 krb5kdc[15973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851242, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:42.870543 16560 init.cc:377] Logged in from keytab as oryx/127.7.146.193@KRBTEST.COM (short username oryx)
I20260619 06:40:42.872304 16560 webserver.cc:460] Webserver started at http://127.7.146.193:37533/ using document root <none> and password file <none>
I20260619 06:40:42.873237 16560 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:42.873378 16560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:42.880326 16560 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 06:40:42.883751 16595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:42.885456 16560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:40:42.885636 16560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "06e1c89602fc4d8daf101a5b0812b387"
format_stamp: "Formatted at 2026-06-19 06:40:40 on dist-test-slave-qfk4"
server_key: "05f7321dbe4c44b4221bd79927e1be77"
server_key_iv: "6029b996dce82945b8255d94d13443d4"
server_key_version: "encryptionkey@0"
I20260619 06:40:42.886943 16560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:42.905947 16560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:42.911607 16560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:42.911907 16560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:42.913092 16560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:42.914721 16560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:42.914822 16560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:42.914927 16560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:42.914996 16560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:42.946946 16560 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:36959
I20260619 06:40:42.947006 16707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:36959 every 8 connection(s)
I20260619 06:40:42.948446 16560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:40:42.949555 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 16560
I20260619 06:40:42.950266 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:44769
--local_ip_for_outbound_sockets=127.7.146.194
--tserver_master_addrs=127.7.146.254:39047
--webserver_port=44209
--webserver_interface=127.7.146.194
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.146.194
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:40:42 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:42 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:40:43.274495 16712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:43.274751 16712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:43.274873 16712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:43.282050 16712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:43.282157 16712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:43.282320 16712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:40:43.292035 16712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.194.keytab
--principal=oryx/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:44769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=44209
--tserver_master_addrs=127.7.146.254:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:43.293404 16712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:43.294616 16712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:43.310273 16719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:43.310575 16718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:43.310377 16721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:43.312140 16712 server_base.cc:1034] running on GCE node
I20260619 06:40:43.312778 16712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:43.313755 16712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:43.314978 16712 hybrid_clock.cc:648] HybridClock initialized: now 1781851243314888 us; error 83 us; skew 500 ppm
I20260619 06:40:43.315306 16712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:43 dist-test-slave-qfk4 krb5kdc[15973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851243, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:43.321424 16712 init.cc:377] Logged in from keytab as oryx/127.7.146.194@KRBTEST.COM (short username oryx)
I20260619 06:40:43.323230 16712 webserver.cc:460] Webserver started at http://127.7.146.194:44209/ using document root <none> and password file <none>
I20260619 06:40:43.324050 16712 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:43.324178 16712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:43.329777 16712 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:40:43.332410 16728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:43.333801 16712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:40:43.333954 16712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "be7f771c5c7049f093657b7d6c9e1983"
format_stamp: "Formatted at 2026-06-19 06:40:40 on dist-test-slave-qfk4"
server_key: "e6750ffb201acbbaebb771a96e7936d7"
server_key_iv: "b1cabf0035acf83e743744f473d58c8e"
server_key_version: "encryptionkey@0"
I20260619 06:40:43.334916 16712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:43.358012 16712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:43.362187 16712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:43.362437 16712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:43.363466 16712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:43.364845 16712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:43.364943 16712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:43.365039 16712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:43.365105 16712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:43.400822 16712 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:44769
I20260619 06:40:43.400878 16840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:44769 every 8 connection(s)
I20260619 06:40:43.402225 16712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:40:43.403403 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 16712
I20260619 06:40:43.404204 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:40467
--local_ip_for_outbound_sockets=127.7.146.195
--tserver_master_addrs=127.7.146.254:39047
--webserver_port=42497
--webserver_interface=127.7.146.195
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.146.195
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:40:43 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:43 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:40:43.737740 16845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:43.738055 16845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:43.738189 16845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:43.745692 16845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:43.745819 16845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:43.746006 16845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:40:43.755688 16845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/krb5kdc/oryx_127.7.146.195.keytab
--principal=oryx/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:40467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=42497
--tserver_master_addrs=127.7.146.254:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:43.757062 16845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:43.758355 16845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:43.773207 16851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:43.785032 16852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:43.786585 16855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:43.786602 16845 server_base.cc:1034] running on GCE node
I20260619 06:40:43.787401 16845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:43.788482 16845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:43.789795 16845 hybrid_clock.cc:648] HybridClock initialized: now 1781851243789717 us; error 62 us; skew 500 ppm
I20260619 06:40:43.790165 16845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:40:43 dist-test-slave-qfk4 krb5kdc[15973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851243, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:40:43.793277 16845 init.cc:377] Logged in from keytab as oryx/127.7.146.195@KRBTEST.COM (short username oryx)
I20260619 06:40:43.795019 16845 webserver.cc:460] Webserver started at http://127.7.146.195:42497/ using document root <none> and password file <none>
I20260619 06:40:43.795722 16845 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:43.795854 16845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:43.801437 16845 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 06:40:43.804080 16861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:43.805471 16845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:40:43.805613 16845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "5990656ac2db46dd872511764d395436"
format_stamp: "Formatted at 2026-06-19 06:40:41 on dist-test-slave-qfk4"
server_key: "0d71b5a0e25f51b56b7ce2ffaefddd60"
server_key_iv: "60ee416859a6ec349c5302b395bce21a"
server_key_version: "encryptionkey@0"
I20260619 06:40:43.806540 16845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:43.836638 16845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:43.841032 16845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:43.841310 16845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:43.842351 16845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:40:43.843761 16845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:40:43.843853 16845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:43.843961 16845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:40:43.844031 16845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:43.878940 16845 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:40467
I20260619 06:40:43.878986 16973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:40467 every 8 connection(s)
I20260619 06:40:43.880604 16845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:40:43.881882 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 16845
Jun 19 06:40:43 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:43 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:43.935259 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:43.883163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.7.146.254:39047)
0619 06:40:43.907815 (+ 24652us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:43.907823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:43.912583 (+ 4760us) server_negotiation.cc:407] Connection header received
0619 06:40:43.912650 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:43.912656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:43.912717 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:43.912824 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:43.915214 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:43.916037 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:43.918206 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:43.919287 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:43.919415 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:43.923566 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:43.923580 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:43.923607 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:43.923668 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:43.928047 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:43.929285 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:43.929298 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:43.929312 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:43.929603 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:40:43.930189 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:40:43.930196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:40:43.930200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:40:43.931033 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:43.931392 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:43.933243 (+ 1851us) server_negotiation.cc:299] Negotiation successful
0619 06:40:43.933847 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":294,"server-negotiator.queue_time_us":24466,"thread_start_us":24668,"threads_started":1}
W20260619 06:40:44.194072 16711 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.231s user 0.000s sys 0.011s
W20260619 06:40:44.195909 16710 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:42.952141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:41061 (local address 127.7.146.254:39047)
0619 06:40:42.952998 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:42.953017 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:42.953522 (+ 505us) server_negotiation.cc:407] Connection header received
0619 06:40:42.956133 (+ 2611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:42.956147 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:42.956540 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:42.956916 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:42.958531 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:42.959814 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:42.961449 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:42.962111 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:42.962286 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.195529 (+1233243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:41061: BlockingRecv error: recv got EOF from 127.7.146.193:41061 (error 108)
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":313,"threads_started":1}
I20260619 06:40:44.196895 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:42.952218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:41061)
0619 06:40:42.953299 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:42.953361 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:42.955498 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:42.957114 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:42.957126 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:42.957696 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:42.958351 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:42.958370 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:42.959991 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:42.959995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:42.961298 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:42.961316 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:42.962334 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:42.962342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:42.962591 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:42.963253 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:42.963276 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:44.195129 (+1231853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":628,"threads_started":1}
W20260619 06:40:44.198201 16708 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:39047 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.146.254:39047: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:44.212108 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:44.199033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:60677)
0619 06:40:44.199334 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:44.199355 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:44.199496 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:44.200069 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:44.200074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:44.200678 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:44.200980 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.200994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.202715 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.202722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.204370 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.204386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.205599 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.205608 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.205812 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.206611 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:44.206637 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:44.211758 (+ 5121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160}
W20260619 06:40:44.212389 16710 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:44.199119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:60677 (local address 127.7.146.254:39047)
0619 06:40:44.199291 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:44.199298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:44.199438 (+ 140us) server_negotiation.cc:407] Connection header received
0619 06:40:44.199623 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:44.199629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:44.199721 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:44.199841 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:44.201179 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.202497 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.204557 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.205324 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.205472 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.212083 (+ 6611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:60677: BlockingRecv error: recv got EOF from 127.7.146.193:60677 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:40:44.232177 16710 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:44.213747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:39633 (local address 127.7.146.254:39047)
0619 06:40:44.213915 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:44.213922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:44.218919 (+ 4997us) server_negotiation.cc:407] Connection header received
0619 06:40:44.219092 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:44.219098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:44.219237 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:44.219361 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:44.220647 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.221925 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.224144 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.224905 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.225018 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.231853 (+ 6835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:39633: BlockingRecv error: recv got EOF from 127.7.146.193:39633 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:40:44.233174 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:44.213325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:39633)
0619 06:40:44.218779 (+ 5454us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:44.218804 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:44.218961 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:44.219597 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:44.219604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:44.220147 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:44.220402 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.220415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.222152 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.222159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.223955 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.223972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.225350 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.225357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.225578 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.226352 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:44.226378 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:44.232715 (+ 6337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5323,"thread_start_us":10613,"threads_started":1}
W20260619 06:40:44.233762 16708 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:40:44.567847 16843 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.144s user 0.004s sys 0.012s
W20260619 06:40:44.569306 16844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:43.406547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:55989 (local address 127.7.146.254:39047)
0619 06:40:43.412994 (+ 6447us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:43.413002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:43.414075 (+ 1073us) server_negotiation.cc:407] Connection header received
0619 06:40:43.415685 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:43.415693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:43.415809 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:43.415951 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:43.417953 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:43.419206 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:43.421390 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:43.422089 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:43.422226 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.568933 (+1146707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:55989: BlockingRecv error: recv got EOF from 127.7.146.194:55989 (error 108)
Metrics: {"server-negotiator.queue_time_us":6308,"thread_start_us":6207,"threads_started":1}
I20260619 06:40:44.569691 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:43.405902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:55989)
0619 06:40:43.413549 (+ 7647us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:43.413598 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:43.415356 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:43.416215 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:43.416228 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:43.416975 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:43.417776 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:43.417799 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:43.419378 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:43.419383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:43.421231 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:43.421245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:43.422343 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:43.422351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:43.422629 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:43.423456 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:43.423494 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:44.568660 (+1145166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":7330,"thread_start_us":7111,"threads_started":1}
W20260619 06:40:44.570554 16841 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:39047 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.146.254:39047: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:44.580885 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:44.571145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:38575)
0619 06:40:44.571429 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:44.571445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:44.571550 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:44.571892 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:44.571894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:44.572210 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:44.572407 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.572418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.574057 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.574061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.575045 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.575060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.576051 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.576055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.576207 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.576793 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:44.576811 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:44.580598 (+ 3787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173}
W20260619 06:40:44.581154 16844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:44.571253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:38575 (local address 127.7.146.254:39047)
0619 06:40:44.571445 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:44.571452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:44.571503 (+ 51us) server_negotiation.cc:407] Connection header received
0619 06:40:44.571658 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:44.571664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:44.571765 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:44.571881 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:44.572621 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.573888 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.575182 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.575859 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.575991 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.580881 (+ 4890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:38575: BlockingRecv error: recv got EOF from 127.7.146.194:38575 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:44.591358 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:44.581861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:58973)
0619 06:40:44.582167 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:44.582182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:44.582263 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:44.582632 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:44.582636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:44.582944 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:44.583101 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.583109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.584631 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.584637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.585610 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.585623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.586676 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.586682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.586841 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.587527 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:44.587544 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:44.591118 (+ 3574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213}
W20260619 06:40:44.591751 16844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:44.582021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:58973 (local address 127.7.146.254:39047)
0619 06:40:44.582217 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:44.582224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:44.582256 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:40:44.582354 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:44.582361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:44.582457 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:44.582569 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:44.583224 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.584450 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.585747 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.586503 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.586631 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.591463 (+ 4832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:58973: BlockingRecv error: recv got EOF from 127.7.146.194:58973 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 06:40:44.591780 16841 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:40:44.974203 16978 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.054s user 0.002s sys 0.016s
W20260619 06:40:44.975518 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:43.892892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:41263 (local address 127.7.146.254:39047)
0619 06:40:43.909902 (+ 17010us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:43.909909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:43.909940 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:40:43.910028 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:43.910035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:43.910139 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:43.910316 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:43.913135 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:43.914529 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:43.917402 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:43.918118 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:43.918215 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.975199 (+1056984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:41263: BlockingRecv error: recv got EOF from 127.7.146.195:41263 (error 108)
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":16652,"thread_start_us":16790,"threads_started":1}
I20260619 06:40:44.976046 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:43.892180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:41263)
0619 06:40:43.895898 (+ 3718us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:43.895973 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:43.898962 (+ 2989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:43.910747 (+ 11785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:43.910766 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:43.911771 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:43.912819 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:43.912919 (+ 100us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:43.914770 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:43.914776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:43.917213 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:43.917231 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:43.918449 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:43.918456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:43.918808 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:43.920266 (+ 1458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:43.920304 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:44.975022 (+1054718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3252,"thread_start_us":2965,"threads_started":1}
W20260619 06:40:44.977033 16974 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:39047 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.146.254:39047: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:44.985620 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:44.977632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:47677)
0619 06:40:44.977839 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:44.977855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:44.977945 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:44.978201 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:44.978204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:44.978534 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:44.978685 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.978693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.979700 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.979704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.980695 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.980709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.981433 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.981437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.981575 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.982126 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:44.982144 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:44.985402 (+ 3258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:40:44.985857 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:44.977698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:47677 (local address 127.7.146.254:39047)
0619 06:40:44.977887 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:44.977892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:44.977908 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:40:44.978034 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:44.978037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:44.978099 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:44.978176 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:44.978830 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.979562 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.980818 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.981254 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.981349 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.985632 (+ 4283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:47677: BlockingRecv error: recv got EOF from 127.7.146.195:47677 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:44 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:44.994194 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:44.986445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:38563)
0619 06:40:44.986615 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:44.986633 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:44.986733 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:44.987037 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:44.987042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:44.987326 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:44.987474 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.987482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.988402 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.988407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.989388 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:44.989401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.990060 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.990066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:44.990247 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.990785 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:44.990801 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:44.993870 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:40:44.994328 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:44.986512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:38563 (local address 127.7.146.254:39047)
0619 06:40:44.986740 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:44.986745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:44.986762 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:40:44.986849 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:44.986854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:44.986917 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:44.986992 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:44.987597 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.988280 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.989519 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:44.989922 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:44.990025 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:44.994083 (+ 4058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:38563: BlockingRecv error: recv got EOF from 127.7.146.195:38563 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
W20260619 06:40:44.994654 16974 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 06:40:45 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:45 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:45.243856 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:45.234323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:40175)
0619 06:40:45.234816 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:45.234839 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:45.234957 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:45.235349 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:45.235384 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:45.235854 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:45.236007 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:45.236017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.237025 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.237028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:45.237985 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:45.237996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.238867 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.238872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:45.239030 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:45.239668 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:45.239688 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:45.243546 (+ 3858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":279,"threads_started":1}
W20260619 06:40:45.243964 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:45.234420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:40175 (local address 127.7.146.254:39047)
0619 06:40:45.234624 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:45.234632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:45.234917 (+ 285us) server_negotiation.cc:407] Connection header received
0619 06:40:45.235063 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:45.235067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:45.235164 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:45.235258 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:45.236165 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.236894 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.238119 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.238596 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.238741 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:45.243745 (+ 5004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:40175: BlockingRecv error: recv got EOF from 127.7.146.193:40175 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 06:40:45 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:45 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:45.600826 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:45.592338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:55483)
0619 06:40:45.592890 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:45.592910 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:45.593052 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:45.593387 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:45.593392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:45.593806 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:45.593950 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:45.593958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.595013 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.595017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:45.595946 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:45.595958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.596687 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.596693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:45.596849 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:45.597365 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:45.597384 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:45.600554 (+ 3170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":291,"threads_started":1}
W20260619 06:40:45.601076 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:45.592438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:55483 (local address 127.7.146.254:39047)
0619 06:40:45.592630 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:45.592636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:45.592987 (+ 351us) server_negotiation.cc:407] Connection header received
0619 06:40:45.593167 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:45.593172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:45.593238 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:45.593320 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:45.594083 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.594867 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.596080 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.596493 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.596644 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:45.600833 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:55483: BlockingRecv error: recv got EOF from 127.7.146.194:55483 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:40:45 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:45 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:46.003855 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:45.995168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:33799)
0619 06:40:45.995642 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:45.995658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:45.995768 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:45.996087 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:45.996090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:45.996494 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:45.996669 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:45.996677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.997713 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.997716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:45.998741 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:45.998753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.999532 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.999537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:45.999648 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:46.000238 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:46.000255 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:46.003628 (+ 3373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":283,"threads_started":1}
W20260619 06:40:46.004065 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:45.995242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:33799 (local address 127.7.146.254:39047)
0619 06:40:45.995456 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:45.995463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:45.995732 (+ 269us) server_negotiation.cc:407] Connection header received
0619 06:40:45.995891 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:45.995894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:45.995972 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:45.996057 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:45.996871 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.997612 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.998898 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:45.999375 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:45.999495 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:46.003844 (+ 4349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:33799: BlockingRecv error: recv got EOF from 127.7.146.195:33799 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 06:40:46 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:46 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:46.253301 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:46.244942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:58767)
0619 06:40:46.245397 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:46.245417 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:46.245536 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:46.245862 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:46.245866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:46.246255 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:46.246442 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:46.246452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:46.247406 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:46.247410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:46.248332 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:46.248343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:46.249074 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:46.249080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:46.249230 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:46.249763 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:46.249846 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:46.253065 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":246,"threads_started":1}
W20260619 06:40:46.253495 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:46.244994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:58767 (local address 127.7.146.254:39047)
0619 06:40:46.245188 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:46.245193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:46.245479 (+ 286us) server_negotiation.cc:407] Connection header received
0619 06:40:46.245665 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:46.245668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:46.245730 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:46.245806 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:46.246577 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:46.247266 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:46.248464 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:46.248933 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:46.249039 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:46.253266 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:58767: BlockingRecv error: recv got EOF from 127.7.146.193:58767 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 06:40:46 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:46 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:46.610904 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:46.601753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:38679)
0619 06:40:46.602230 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:46.602248 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:46.602380 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:46.602681 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:46.602685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:46.603113 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:46.603339 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:46.603364 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:46.604321 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:46.604326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:46.605277 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:46.605289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:46.605963 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:46.605969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:46.606144 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:46.606791 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:46.606812 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:46.610528 (+ 3716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":263,"threads_started":1}
W20260619 06:40:46.610991 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:46.601908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:38679 (local address 127.7.146.254:39047)
0619 06:40:46.602124 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:46.602130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:46.602330 (+ 200us) server_negotiation.cc:407] Connection header received
0619 06:40:46.602492 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:46.602495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:46.602559 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:46.602640 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:46.603480 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:46.604191 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:46.605410 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:46.605826 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:46.605939 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:46.610791 (+ 4852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:38679: BlockingRecv error: recv got EOF from 127.7.146.194:38679 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 06:40:47 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:47 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:47.013761 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:47.004817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:44171)
0619 06:40:47.005261 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:47.005281 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:47.005448 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:47.005821 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:47.005826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:47.006231 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:47.006419 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:47.006429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.007468 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.007473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:47.008554 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:47.008596 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.009334 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.009340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:47.009510 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:47.010170 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:47.010193 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:47.013493 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
W20260619 06:40:47.014075 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:47.005064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:44171 (local address 127.7.146.254:39047)
0619 06:40:47.005264 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:47.005271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:47.005350 (+ 79us) server_negotiation.cc:407] Connection header received
0619 06:40:47.005574 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:47.005579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:47.005657 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:47.005741 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:47.006565 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.007328 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.008714 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.009194 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.009309 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:47.013778 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:44171: BlockingRecv error: recv got EOF from 127.7.146.195:44171 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 06:40:47 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:47 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:47.263778 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:47.254303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:47215)
0619 06:40:47.254842 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:47.254864 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:47.255012 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:47.255369 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:47.255375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:47.255768 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:47.255951 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:47.255963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.257123 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.257128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:47.258076 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:47.258089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.258826 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.258832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:47.259015 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:47.259632 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:47.259650 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:47.263462 (+ 3812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
W20260619 06:40:47.263927 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:47.254288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:47215 (local address 127.7.146.254:39047)
0619 06:40:47.254451 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:47.254456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:47.254968 (+ 512us) server_negotiation.cc:407] Connection header received
0619 06:40:47.255144 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:47.255148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:47.255219 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:47.255314 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:47.256116 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.256973 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.258225 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.258669 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.258799 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:47.263716 (+ 4917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:47215: BlockingRecv error: recv got EOF from 127.7.146.193:47215 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 06:40:47 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:47 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:47.620810 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:47.611889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:58081)
0619 06:40:47.612381 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:47.612397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:47.612505 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:47.612938 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:47.612941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:47.613262 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:47.613426 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:47.613434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.614572 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.614577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:47.615540 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:47.615554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.616366 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.616372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:47.616501 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:47.617088 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:47.617106 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:47.620541 (+ 3435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":234,"threads_started":1}
W20260619 06:40:47.620955 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:47.611943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:58081 (local address 127.7.146.254:39047)
0619 06:40:47.612107 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:47.612111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:47.612469 (+ 358us) server_negotiation.cc:407] Connection header received
0619 06:40:47.612660 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:47.612664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:47.612784 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:47.612880 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:47.613570 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.614416 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.615690 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:47.616218 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:47.616319 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:47.620762 (+ 4443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:58081: BlockingRecv error: recv got EOF from 127.7.146.194:58081 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 06:40:48 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:48 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:48.023186 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:48.014700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:50601)
0619 06:40:48.015241 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:48.015258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:48.015370 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:48.015684 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:48.015687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:48.016069 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:48.016257 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:48.016266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.017445 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.017450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:48.018339 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:48.018352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.019003 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.019008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:48.019135 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:48.019703 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:48.019726 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:48.022937 (+ 3211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":273,"threads_started":1}
W20260619 06:40:48.023301 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:48.015020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:50601 (local address 127.7.146.254:39047)
0619 06:40:48.015213 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:48.015220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:48.015389 (+ 169us) server_negotiation.cc:407] Connection header received
0619 06:40:48.015477 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:48.015482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:48.015563 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:48.015651 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:48.016413 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.017277 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.018481 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.018876 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.018966 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:48.023118 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:50601: BlockingRecv error: recv got EOF from 127.7.146.195:50601 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 06:40:48 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:48 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:48.274020 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:48.264960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:38329)
0619 06:40:48.265490 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:48.265514 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:48.265693 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:48.266071 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:48.266074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:48.266847 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:48.267060 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:48.267073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.268208 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.268214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:48.269253 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:48.269265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.269904 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.269910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:48.270074 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:48.270573 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:48.270590 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:48.273670 (+ 3080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":263,"threads_started":1}
W20260619 06:40:48.274109 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:48.265016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:38329 (local address 127.7.146.254:39047)
0619 06:40:48.265256 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:48.265263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:48.265595 (+ 332us) server_negotiation.cc:407] Connection header received
0619 06:40:48.265842 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:48.265848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:48.265931 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:48.266033 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:48.267179 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.268084 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.269378 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.269776 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.269878 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:48.273880 (+ 4002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:38329: BlockingRecv error: recv got EOF from 127.7.146.193:38329 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 06:40:48 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:48 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:48.630192 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:48.621730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:44267)
0619 06:40:48.622204 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:48.622221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:48.622340 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:48.622665 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:48.622670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:48.623058 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:48.623223 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:48.623231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.624198 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.624202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:48.625210 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:48.625221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.625867 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.625873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:48.626029 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:48.626636 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:48.626655 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:48.629957 (+ 3302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":252,"threads_started":1}
W20260619 06:40:48.630327 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:48.621838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:44267 (local address 127.7.146.254:39047)
0619 06:40:48.622021 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:48.622028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:48.622293 (+ 265us) server_negotiation.cc:407] Connection header received
0619 06:40:48.622466 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:48.622471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:48.622530 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:48.622607 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:48.623368 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.624072 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.625334 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:48.625739 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:48.625843 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:48.630125 (+ 4282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:44267: BlockingRecv error: recv got EOF from 127.7.146.194:44267 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 06:40:49 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:49 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:49.033130 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:49.024201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:32779)
0619 06:40:49.024663 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:49.024679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:49.024810 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:49.025149 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:49.025153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:49.025528 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:49.025689 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:49.025699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.026964 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.026970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:49.028126 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:49.028139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.028883 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.028888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:49.029017 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:49.029575 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:49.029591 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:49.032896 (+ 3305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
W20260619 06:40:49.033295 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:49.024296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:32779 (local address 127.7.146.254:39047)
0619 06:40:49.024531 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:49.024536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:49.024751 (+ 215us) server_negotiation.cc:407] Connection header received
0619 06:40:49.024909 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:49.024913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:49.024997 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:49.025075 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:49.025837 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.026825 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.028243 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.028754 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.028840 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:49.033069 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:32779: BlockingRecv error: recv got EOF from 127.7.146.195:32779 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 06:40:49 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:49 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:49.283772 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:49.274915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:35647)
0619 06:40:49.275389 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:49.275408 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:49.275521 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:49.275824 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:49.275827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:49.276223 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:49.276378 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:49.276387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.277724 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.277731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:49.278690 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:49.278705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.279413 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.279419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:49.279588 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:49.280130 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:49.280149 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:49.283491 (+ 3342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
W20260619 06:40:49.283864 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:49.275026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:35647 (local address 127.7.146.254:39047)
0619 06:40:49.275194 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:49.275199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:49.275501 (+ 302us) server_negotiation.cc:407] Connection header received
0619 06:40:49.275619 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:49.275623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:49.275702 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:49.275802 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:49.276520 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.277587 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.278844 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.279270 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.279384 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:49.283668 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:35647: BlockingRecv error: recv got EOF from 127.7.146.193:35647 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 06:40:49 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:49 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:49.639437 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:49.631008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:51045)
0619 06:40:49.631475 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:49.631491 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:49.631601 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:49.631874 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:49.631878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:49.632259 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:49.632398 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:49.632407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.633393 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.633398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:49.634364 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:49.634376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.635024 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.635028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:49.635146 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:49.635659 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:49.635677 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:49.639181 (+ 3504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":246,"threads_started":1}
W20260619 06:40:49.639575 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:49.631151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:51045 (local address 127.7.146.254:39047)
0619 06:40:49.631292 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:49.631296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:49.631555 (+ 259us) server_negotiation.cc:407] Connection header received
0619 06:40:49.631693 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:49.631697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:49.631758 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:49.631835 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:49.632505 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.633263 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.634470 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:49.634911 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:49.635018 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:49.639346 (+ 4328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:51045: BlockingRecv error: recv got EOF from 127.7.146.194:51045 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 06:40:50 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:50 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:50.042872 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:50.034022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:52023)
0619 06:40:50.034511 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:50.034527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:50.034666 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:50.035063 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:50.035066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:50.035462 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:50.035601 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:50.035609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.036661 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.036665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:50.037572 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:50.037584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.038390 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.038395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:50.038637 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:50.039238 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:50.039260 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:50.042641 (+ 3381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":239,"threads_started":1}
W20260619 06:40:50.043081 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:50.034094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:52023 (local address 127.7.146.254:39047)
0619 06:40:50.034249 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:50.034253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:50.034647 (+ 394us) server_negotiation.cc:407] Connection header received
0619 06:40:50.034812 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:50.034817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:50.034940 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:50.035035 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:50.035778 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.036508 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.037704 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.038208 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.038327 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:50.042843 (+ 4516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:52023: BlockingRecv error: recv got EOF from 127.7.146.195:52023 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 06:40:50 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:50 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:50.293347 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:50.284919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:47905)
0619 06:40:50.285378 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:50.285403 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:50.285531 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:50.285837 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:50.285841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:50.286245 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:50.286401 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:50.286410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.287391 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.287396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:50.288321 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:50.288333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.289098 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.289104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:50.289270 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:50.289806 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:50.289823 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:50.293031 (+ 3208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":237,"threads_started":1}
W20260619 06:40:50.293503 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:50.285050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:47905 (local address 127.7.146.254:39047)
0619 06:40:50.285221 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:50.285225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:50.285485 (+ 260us) server_negotiation.cc:407] Connection header received
0619 06:40:50.285638 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:50.285644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:50.285714 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:50.285790 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:50.286528 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.287253 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.288451 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.288955 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.289060 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:50.293259 (+ 4199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:47905: BlockingRecv error: recv got EOF from 127.7.146.193:47905 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 06:40:50 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:50 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:50.648402 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:50.640247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:42951)
0619 06:40:50.640717 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:50.640734 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:50.640846 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:50.641102 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:50.641105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:50.641497 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:50.641653 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:50.641662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.642614 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.642618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:50.643544 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:50.643556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.644238 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.644242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:50.644366 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:50.644892 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:50.644908 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:50.648155 (+ 3247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":228,"threads_started":1}
W20260619 06:40:50.648547 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:50.640311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:42951 (local address 127.7.146.254:39047)
0619 06:40:50.640527 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:50.640534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:50.640827 (+ 293us) server_negotiation.cc:407] Connection header received
0619 06:40:50.640934 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:50.640938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:50.640999 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:50.641075 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:50.641793 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.642486 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.643680 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:50.644104 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:50.644209 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:50.648371 (+ 4162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:42951: BlockingRecv error: recv got EOF from 127.7.146.194:42951 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 06:40:51 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:51 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:51.052768 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:51.043801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:48753)
0619 06:40:51.044263 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:51.044279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:51.044430 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:51.044789 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:51.044794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:51.045205 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:51.045378 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:51.045387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.046437 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.046441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:51.047336 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:51.047359 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.048020 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.048026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:51.048188 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:51.048783 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:51.048801 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:51.052486 (+ 3685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":239,"threads_started":1}
W20260619 06:40:51.052927 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:51.043890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:48753 (local address 127.7.146.254:39047)
0619 06:40:51.044094 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:51.044101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:51.044359 (+ 258us) server_negotiation.cc:407] Connection header received
0619 06:40:51.044542 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:51.044547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:51.044658 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:51.044741 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:51.045513 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.046285 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.047477 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.047875 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.047978 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:51.052700 (+ 4722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:48753: BlockingRecv error: recv got EOF from 127.7.146.195:48753 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 06:40:51 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:51 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:51.303395 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:51.294473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:51669)
0619 06:40:51.295001 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:51.295022 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:51.295137 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:51.295473 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:51.295477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:51.295867 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:51.296034 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:51.296042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.297070 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.297075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:51.298132 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:51.298146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.298963 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.298968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:51.299086 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:51.299633 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:51.299649 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:51.303099 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":249,"threads_started":1}
W20260619 06:40:51.303464 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:51.294526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:51669 (local address 127.7.146.254:39047)
0619 06:40:51.294711 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:51.294719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:51.295097 (+ 378us) server_negotiation.cc:407] Connection header received
0619 06:40:51.295271 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:51.295275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:51.295343 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:51.295429 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:51.296170 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.296912 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.298300 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.298811 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.298909 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:51.303286 (+ 4377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:51669: BlockingRecv error: recv got EOF from 127.7.146.193:51669 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:40:51 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:51 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:51.658479 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:51.649405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:53771)
0619 06:40:51.649969 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:51.649987 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:51.650115 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:51.650468 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:51.650472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:51.650852 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:51.651016 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:51.651024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.652049 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.652052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:51.653322 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:51.653338 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.654051 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.654057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:51.654170 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:51.654788 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:51.654806 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:51.658219 (+ 3413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":262,"threads_started":1}
W20260619 06:40:51.658779 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:51.649506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:53771 (local address 127.7.146.254:39047)
0619 06:40:51.649720 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:51.649724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:51.650091 (+ 367us) server_negotiation.cc:407] Connection header received
0619 06:40:51.650264 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:51.650268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:51.650338 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:51.650434 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:51.651163 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.651931 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.653479 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:51.653903 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:51.654002 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:51.658448 (+ 4446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:53771: BlockingRecv error: recv got EOF from 127.7.146.194:53771 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Jun 19 06:40:52 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:52 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:52.062724 17002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:52.053816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:46661)
0619 06:40:52.054400 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:52.054415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:52.054546 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:52.054872 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:52.054875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:52.055258 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:52.055435 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:52.055444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.056742 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.056746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:52.057826 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:52.057849 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.058569 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.058573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:52.058680 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:52.059260 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:52.059276 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:52.062495 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":280,"threads_started":1}
W20260619 06:40:52.063021 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:52.053942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:46661 (local address 127.7.146.254:39047)
0619 06:40:52.054123 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:52.054127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:52.054496 (+ 369us) server_negotiation.cc:407] Connection header received
0619 06:40:52.054645 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:52.054649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:52.054720 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:52.054851 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:52.055606 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.056606 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.057975 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.058444 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.058557 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:52.062692 (+ 4135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:46661: BlockingRecv error: recv got EOF from 127.7.146.195:46661 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 06:40:52 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:52 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:52.313284 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:52.304469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:37995)
0619 06:40:52.305056 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:52.305075 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:52.305201 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:52.305490 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:52.305492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:52.305870 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:52.306028 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:52.306037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.307022 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.307025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:52.307950 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:52.307961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.308676 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.308681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:52.308803 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:52.309379 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:52.309397 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:52.312945 (+ 3548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":337,"threads_started":1}
W20260619 06:40:52.313369 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:52.304681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:37995 (local address 127.7.146.254:39047)
0619 06:40:52.304878 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:52.304883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:52.305141 (+ 258us) server_negotiation.cc:407] Connection header received
0619 06:40:52.305294 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:52.305298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:52.305362 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:52.305452 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:52.306153 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.306910 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.308068 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.308480 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.308617 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:52.313150 (+ 4533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:37995: BlockingRecv error: recv got EOF from 127.7.146.193:37995 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 06:40:52 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:52 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:52.668366 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:52.659378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:35657)
0619 06:40:52.659816 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:52.659831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:52.659970 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:52.660242 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:52.660245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:52.660665 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:52.660871 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:52.660880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.662459 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.662463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:52.663376 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:52.663387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.664169 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.664175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:52.664326 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:52.664912 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:52.664929 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:52.668130 (+ 3201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":220,"threads_started":1}
W20260619 06:40:52.668601 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:52.659476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:35657 (local address 127.7.146.254:39047)
0619 06:40:52.659718 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:52.659725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:52.659968 (+ 243us) server_negotiation.cc:407] Connection header received
0619 06:40:52.660059 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:52.660062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:52.660130 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:52.660217 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:52.661014 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.662327 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.663500 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:52.664022 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:52.664111 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:52.668357 (+ 4246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:35657: BlockingRecv error: recv got EOF from 127.7.146.194:35657 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 06:40:53 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:53 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:53.072047 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:53.063724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:48325)
0619 06:40:53.064238 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:53.064255 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:53.064367 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:53.064652 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:53.064655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:53.065032 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:53.065262 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:53.065276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.066220 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.066224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:53.067151 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:53.067163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.067815 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.067819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:53.067927 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:53.068478 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:53.068495 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:53.071810 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":260,"threads_started":1}
W20260619 06:40:53.072304 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:53.063772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:48325 (local address 127.7.146.254:39047)
0619 06:40:53.063924 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:53.063928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:53.064322 (+ 394us) server_negotiation.cc:407] Connection header received
0619 06:40:53.064454 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:53.064457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:53.064517 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:53.064633 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:53.065393 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.066099 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.067287 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.067685 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.067773 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:53.072047 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:48325: BlockingRecv error: recv got EOF from 127.7.146.195:48325 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 06:40:53 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:53 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:53.322997 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:53.314430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:60109)
0619 06:40:53.314931 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:53.314951 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:53.315090 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:53.315455 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:53.315460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:53.315959 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:53.316143 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:53.316156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.317295 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.317298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:53.318218 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:53.318229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.318893 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.318897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:53.319012 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:53.319587 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:53.319604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:53.322764 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":276,"threads_started":1}
W20260619 06:40:53.323254 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:53.314570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:60109 (local address 127.7.146.254:39047)
0619 06:40:53.314794 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:53.314800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:53.315041 (+ 241us) server_negotiation.cc:407] Connection header received
0619 06:40:53.315211 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:53.315216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:53.315293 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:53.315400 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:53.316268 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.317182 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.318343 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.318757 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.318847 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:53.323020 (+ 4173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:60109: BlockingRecv error: recv got EOF from 127.7.146.193:60109 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 06:40:53 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:53 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:53.683395 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:53.669441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:50789)
0619 06:40:53.669948 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:53.669967 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:53.670085 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:53.670367 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:53.670371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:53.670791 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:53.670951 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:53.670960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.671927 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.671931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:53.672961 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:53.672975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.673696 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.673700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:53.673873 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:53.679570 (+ 5697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:53.679590 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:53.683061 (+ 3471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
W20260619 06:40:53.683504 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:53.669450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:50789 (local address 127.7.146.254:39047)
0619 06:40:53.669599 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:53.669604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:53.670037 (+ 433us) server_negotiation.cc:407] Connection header received
0619 06:40:53.670175 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:53.670179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:53.670237 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:53.670314 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:53.671064 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.671765 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.673106 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:53.673501 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:53.673589 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:53.683266 (+ 9677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:50789: BlockingRecv error: recv got EOF from 127.7.146.194:50789 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 06:40:54 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:54 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:54.082219 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:54.073199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:41187)
0619 06:40:54.073730 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:54.073750 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:54.073890 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:54.074238 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:54.074243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:54.074749 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:54.074897 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:54.074906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.075853 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.075857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:54.077008 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:54.077023 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.077744 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.077750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:54.077887 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:54.078494 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:54.078514 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:54.081990 (+ 3476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":281,"threads_started":1}
W20260619 06:40:54.082389 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:54.073264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:41187 (local address 127.7.146.254:39047)
0619 06:40:54.073482 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:54.073488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:54.073863 (+ 375us) server_negotiation.cc:407] Connection header received
0619 06:40:54.074007 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:54.074012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:54.074092 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:54.074183 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:54.075026 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.075706 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.077128 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.077543 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.077630 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:54.082186 (+ 4556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:41187: BlockingRecv error: recv got EOF from 127.7.146.195:41187 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 06:40:54 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:54 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:54.332764 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:54.323880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:58129)
0619 06:40:54.324363 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:54.324382 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:54.324510 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:54.324859 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:54.324864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:54.325222 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:54.325380 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:54.325389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.326357 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.326362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:54.327393 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:54.327408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.328177 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.328183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:54.328342 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:54.328917 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:54.328934 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:54.332444 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":242,"threads_started":1}
W20260619 06:40:54.332893 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:54.323989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:58129 (local address 127.7.146.254:39047)
0619 06:40:54.324242 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:54.324248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:54.324461 (+ 213us) server_negotiation.cc:407] Connection header received
0619 06:40:54.324643 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:54.324648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:54.324710 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:54.324788 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:54.325517 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.326224 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.327532 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.328022 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.328146 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:54.332649 (+ 4503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:58129: BlockingRecv error: recv got EOF from 127.7.146.193:58129 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 06:40:54 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:54 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:54.692775 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:54.684278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:47577)
0619 06:40:54.684772 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:54.684788 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:54.684950 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:54.685254 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:54.685260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:54.685636 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:54.685789 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:54.685798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.686797 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.686801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:54.687771 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:54.687782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.688446 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.688453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:54.688622 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:54.689193 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:54.689213 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:54.692437 (+ 3224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":270,"threads_started":1}
W20260619 06:40:54.692899 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:54.684369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:47577 (local address 127.7.146.254:39047)
0619 06:40:54.684592 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:54.684598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:54.684904 (+ 306us) server_negotiation.cc:407] Connection header received
0619 06:40:54.685061 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:54.685066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:54.685129 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:54.685205 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:54.685950 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.686657 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.687916 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:54.688297 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:54.688410 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:54.692664 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:47577: BlockingRecv error: recv got EOF from 127.7.146.194:47577 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:40:55 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:55 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:55.091666 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:55.083065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:52327)
0619 06:40:55.083558 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:55.083573 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:55.083694 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:55.084074 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:55.084079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:55.084463 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:55.084643 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:55.084653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.085637 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.085643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:55.086620 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:55.086632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.087365 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.087371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:55.087526 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:55.088082 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:55.088098 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:55.091379 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":269,"threads_started":1}
W20260619 06:40:55.091876 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:55.083245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:52327 (local address 127.7.146.254:39047)
0619 06:40:55.083437 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:55.083443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:55.083648 (+ 205us) server_negotiation.cc:407] Connection header received
0619 06:40:55.083807 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:55.083812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:55.083939 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:55.084022 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:55.084785 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.085495 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.086758 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.087197 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.087303 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:55.091599 (+ 4296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:52327: BlockingRecv error: recv got EOF from 127.7.146.195:52327 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 06:40:55 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:55 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:55.343161 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:55.333800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:46927)
0619 06:40:55.334290 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:55.334313 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:55.334447 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:55.334931 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:55.334936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:55.335437 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:55.335648 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:55.335661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.337020 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.337025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:55.337971 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:55.337984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.338728 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.338735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:55.338874 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:55.339472 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:55.339491 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:55.342934 (+ 3443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":271,"threads_started":1}
W20260619 06:40:55.343297 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:55.333841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:46927 (local address 127.7.146.254:39047)
0619 06:40:55.334025 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:55.334030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:55.334435 (+ 405us) server_negotiation.cc:407] Connection header received
0619 06:40:55.334668 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:55.334673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:55.334758 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:55.334874 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:55.335805 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.336889 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.338107 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.338563 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.338662 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:55.343118 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:46927: BlockingRecv error: recv got EOF from 127.7.146.193:46927 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 06:40:55 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:55 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:55.702633 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:55.693778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:43781)
0619 06:40:55.694434 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:55.694451 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:55.694587 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:55.694927 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:55.694932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:55.695384 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:55.695565 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:55.695576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.696670 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.696675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:55.697615 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:55.697627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.698351 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.698358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:55.698532 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:55.699143 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:55.699169 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:55.702421 (+ 3252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":433,"threads_started":1}
W20260619 06:40:55.702955 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:55.693930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:43781 (local address 127.7.146.254:39047)
0619 06:40:55.694121 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:55.694128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:55.694538 (+ 410us) server_negotiation.cc:407] Connection header received
0619 06:40:55.694712 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:55.694718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:55.694790 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:55.694878 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:55.695717 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.696496 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.697769 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:55.698197 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:55.698318 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:55.702630 (+ 4312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:43781: BlockingRecv error: recv got EOF from 127.7.146.194:43781 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 06:40:56 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:56 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:56.101462 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:56.092872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:54235)
0619 06:40:56.093353 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:56.093373 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:56.093501 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:56.093809 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:56.093814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:56.094216 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:56.094401 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:56.094411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.095400 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.095405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:56.096327 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:56.096339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.097065 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.097071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:56.097239 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:56.097863 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:56.097879 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:56.101243 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":255,"threads_started":1}
W20260619 06:40:56.101619 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:56.093287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:54235 (local address 127.7.146.254:39047)
0619 06:40:56.093445 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:56.093450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:56.093468 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:40:56.093615 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:56.093620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:56.093684 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:56.093763 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:56.094540 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.095276 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.096457 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.096923 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.097031 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:56.101426 (+ 4395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:54235: BlockingRecv error: recv got EOF from 127.7.146.195:54235 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 06:40:56 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:56 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:56.352885 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:56.344117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:35881)
0619 06:40:56.344675 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:56.344694 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:56.344823 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:56.345155 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:56.345160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:56.345570 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:56.345727 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:56.345736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.346754 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.346758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:56.347802 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:56.347818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.348659 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.348666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:56.348847 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:56.349413 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:56.349428 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:56.352637 (+ 3209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":295,"threads_started":1}
W20260619 06:40:56.353135 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:56.344194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:35881 (local address 127.7.146.254:39047)
0619 06:40:56.344482 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:56.344488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:56.344772 (+ 284us) server_negotiation.cc:407] Connection header received
0619 06:40:56.344943 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:56.344948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:56.345014 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:56.345092 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:56.345871 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.346626 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.347965 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.348469 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.348606 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:56.352890 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:35881: BlockingRecv error: recv got EOF from 127.7.146.193:35881 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 19 06:40:56 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:56 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:56.712473 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:56.703494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:58653)
0619 06:40:56.704110 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:56.704131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:56.704281 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:56.704674 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:56.704678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:56.705089 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:56.705252 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:56.705262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.706307 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.706312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:56.707305 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:56.707319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.708013 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.708018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:56.708150 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:56.708774 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:56.708792 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:56.712228 (+ 3436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":372,"threads_started":1}
W20260619 06:40:56.712706 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:56.703570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:58653 (local address 127.7.146.254:39047)
0619 06:40:56.703758 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:56.703764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:56.704219 (+ 455us) server_negotiation.cc:407] Connection header received
0619 06:40:56.704439 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:56.704443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:56.704510 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:56.704651 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:56.705407 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.706151 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.707440 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:56.707841 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:56.707934 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:56.712427 (+ 4493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:58653: BlockingRecv error: recv got EOF from 127.7.146.194:58653 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 06:40:57 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:57 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:57.111191 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:57.102463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:59467)
0619 06:40:57.103063 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:57.103078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:57.103212 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:57.103552 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:57.103557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:57.103949 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:57.104114 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:57.104123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.105167 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.105172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:57.106101 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:57.106114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.106834 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.106841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:57.107006 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:57.107563 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:57.107579 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:57.110943 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":354,"threads_started":1}
W20260619 06:40:57.111446 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:57.102597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:59467 (local address 127.7.146.254:39047)
0619 06:40:57.102823 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:57.102830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:57.103164 (+ 334us) server_negotiation.cc:407] Connection header received
0619 06:40:57.103331 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:57.103336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:57.103400 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:57.103480 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:57.104257 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.105027 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.106247 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.106649 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.106755 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:57.111157 (+ 4402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:59467: BlockingRecv error: recv got EOF from 127.7.146.195:59467 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 06:40:57 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:57 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:57.362645 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:57.353844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:52719)
0619 06:40:57.354324 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:57.354345 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:57.354471 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:57.354783 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:57.354788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:57.355200 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:57.355348 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:57.355357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.356312 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.356316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:57.357260 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:57.357272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.357962 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.357967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:57.358124 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:57.358668 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:57.358685 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:57.362334 (+ 3649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":232,"threads_started":1}
W20260619 06:40:57.362766 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:57.353973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:52719 (local address 127.7.146.254:39047)
0619 06:40:57.354198 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:57.354204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:57.354419 (+ 215us) server_negotiation.cc:407] Connection header received
0619 06:40:57.354579 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:57.354585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:57.354653 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:57.354742 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:57.355482 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.356169 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.357420 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.357832 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.357935 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:57.362547 (+ 4612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:52719: BlockingRecv error: recv got EOF from 127.7.146.193:52719 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:40:57 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:57 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:57.721884 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:57.713368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:60971)
0619 06:40:57.713857 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:57.713873 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:57.713997 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:57.714301 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:57.714306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:57.714725 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:57.714862 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:57.714870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.715843 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.715847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:57.716777 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:57.716788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.717517 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.717523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:57.717678 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:57.718186 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:57.718205 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:57.721597 (+ 3392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":229,"threads_started":1}
W20260619 06:40:57.722070 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:57.713522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:60971 (local address 127.7.146.254:39047)
0619 06:40:57.713718 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:57.713724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:57.713950 (+ 226us) server_negotiation.cc:407] Connection header received
0619 06:40:57.714103 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:57.714109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:57.714177 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:57.714265 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:57.714995 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.715714 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.716908 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:57.717383 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:57.717487 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:57.721838 (+ 4351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:60971: BlockingRecv error: recv got EOF from 127.7.146.194:60971 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 06:40:58 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:58 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.195@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:58.121320 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:58.112181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.195:46509)
0619 06:40:58.112804 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:58.112824 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:58.112965 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:58.113364 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:58.113369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:58.113953 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:58.114110 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:58.114119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.115160 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.115164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:58.116066 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:58.116078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.116813 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.116818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:58.116942 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:58.117523 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:58.117542 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:58.121100 (+ 3558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":314,"threads_started":1}
W20260619 06:40:58.121588 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:58.112301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:46509 (local address 127.7.146.254:39047)
0619 06:40:58.112490 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:58.112496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:58.112921 (+ 425us) server_negotiation.cc:407] Connection header received
0619 06:40:58.113088 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:58.113094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:58.113182 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:58.113286 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:58.114244 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.115028 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.116210 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.116683 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.116785 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:58.121344 (+ 4559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.195:46509: BlockingRecv error: recv got EOF from 127.7.146.195:46509 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 06:40:58 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:58 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.193@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:58.372875 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:58.363953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.193:33613)
0619 06:40:58.364470 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:58.364492 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:58.364692 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:58.365109 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:58.365114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:58.365607 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:58.365858 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:58.365872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.366952 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.366956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:58.367885 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:58.367897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.368534 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.368538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:58.368696 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:58.369221 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:58.369237 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:58.372456 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":269,"threads_started":1}
W20260619 06:40:58.372948 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:58.364053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:33613 (local address 127.7.146.254:39047)
0619 06:40:58.364319 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:58.364325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:58.364607 (+ 282us) server_negotiation.cc:407] Connection header received
0619 06:40:58.364864 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:58.364868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:58.364945 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:58.365032 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:58.366004 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.366800 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.368026 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.368421 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.368505 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:58.372689 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.193:33613: BlockingRecv error: recv got EOF from 127.7.146.193:33613 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Jun 19 06:40:58 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:40:58 dist-test-slave-qfk4 krb5kdc[15973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.146.194@KRBTEST.COM for oryx/127.7.146.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:40:58.735363 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:58.723147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:39047 (local address 127.7.146.194:52571)
0619 06:40:58.723628 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 06:40:58.723645 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:40:58.723773 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:40:58.724148 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:40:58.724153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:40:58.724529 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:40:58.724795 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:58.724805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.725806 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.725811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:58.726759 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:40:58.726772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.727583 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.727589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:40:58.727816 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:58.731407 (+ 3591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:40:58.731429 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:40:58.735078 (+ 3649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.146.254:39047: Server oryx/127.7.146.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":247,"threads_started":1}
W20260619 06:40:58.735576 16979 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:40:58.723329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:52571 (local address 127.7.146.254:39047)
0619 06:40:58.723536 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:58.723542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:58.723729 (+ 187us) server_negotiation.cc:407] Connection header received
0619 06:40:58.723873 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:58.723877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:58.723971 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:58.724053 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:58.724920 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.725613 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.726892 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.727425 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.727540 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:58.735325 (+ 7785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.146.194:52571: BlockingRecv error: recv got EOF from 127.7.146.194:52571 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:40:58.886771 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 16560
I20260619 06:40:58.899433 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:58.888003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34070 (local address 127.7.146.193:36959)
0619 06:40:58.888632 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:58.888642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:58.888670 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:40:58.888738 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:58.888745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:58.888978 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:58.889157 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:58.890546 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.891727 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.893074 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.893669 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.893766 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:58.896229 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:58.896238 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:58.896253 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:58.896297 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:58.898881 (+ 2584us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.146.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 06:40:58.899212 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34070: Request ticket server kudu/127.7.146.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":311,"threads_started":1}
W20260619 06:40:58.899529 17024 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34070: Request ticket server kudu/127.7.146.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 06:40:58.899622 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 16560: Not authorized: Client connection negotiation failed: client connection to 127.7.146.193:36959: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.146.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 06:40:58.899860 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 16560
I20260619 06:40:58.913640 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 16712
I20260619 06:40:58.926522 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:58.914817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.7.146.194:44769)
0619 06:40:58.915447 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:58.915454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:58.915482 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:40:58.915551 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:58.915557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:58.915852 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:58.916042 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:58.917662 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.918741 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.920027 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.920786 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.920886 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:58.923431 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:58.923440 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:58.923455 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:58.923494 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:58.926027 (+ 2533us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.146.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 06:40:58.926285 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40362: Request ticket server kudu/127.7.146.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":304,"threads_started":1}
W20260619 06:40:58.926620 17025 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40362: Request ticket server kudu/127.7.146.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 06:40:58.926723 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 16712: Not authorized: Client connection negotiation failed: client connection to 127.7.146.194:44769: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.146.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 06:40:58.926885 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 16712
I20260619 06:40:58.939826 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 16845
I20260619 06:40:58.951704 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:58.941095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.7.146.195:40467)
0619 06:40:58.941740 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:58.941749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:58.941778 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:40:58.941856 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:58.941865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:58.942166 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:58.942395 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:58.943592 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.944683 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.945777 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:58.946305 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:58.946398 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:58.948706 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:58.948714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:58.948728 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:40:58.948762 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:58.951174 (+ 2412us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.146.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 06:40:58.951464 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49730: Request ticket server kudu/127.7.146.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":297,"threads_started":1}
W20260619 06:40:58.951790 17026 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49730: Request ticket server kudu/127.7.146.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 06:40:58.951857 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 16845: Not authorized: Client connection negotiation failed: client connection to 127.7.146.195:40467: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.146.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 06:40:58.952023 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 16845
I20260619 06:40:58.965161 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 16491
I20260619 06:40:58.966544 16553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:40:59.079849 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 16491
2026-06-19T06:40:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20843 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T06:40:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:40:59Z Disabled control of system clock
I20260619 06:40:59.149396 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38211
--webserver_interface=127.7.146.254
--webserver_port=0
--builtin_ntp_servers=127.7.146.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:38211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:40:59.465094 17031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:40:59.465391 17031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:40:59.465468 17031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:40:59.472622 17031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:40:59.472739 17031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:40:59.472775 17031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:40:59.472826 17031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:40:59.472867 17031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:40:59.482075 17031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:38211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:40:59.483335 17031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:40:59.485090 17031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:40:59.496766 17037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:59.496801 17036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:40:59.496833 17039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:40:59.497227 17031 server_base.cc:1034] running on GCE node
I20260619 06:40:59.498101 17031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:40:59.499440 17031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:40:59.500643 17031 hybrid_clock.cc:648] HybridClock initialized: now 1781851259500597 us; error 40 us; skew 500 ppm
I20260619 06:40:59.501003 17031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:40:59.503144 17031 webserver.cc:460] Webserver started at http://127.7.146.254:45497/ using document root <none> and password file <none>
I20260619 06:40:59.503782 17031 fs_manager.cc:359] Metadata directory not provided
I20260619 06:40:59.503876 17031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:40:59.504154 17031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:40:59.507050 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "74e482e505174fd696ac7f4c7c5c55b2"
format_stamp: "Formatted at 2026-06-19 06:40:59 on dist-test-slave-qfk4"
server_key: "dac8fb83efa701308b74c1fa99885e12"
server_key_iv: "1edf7a918b1ce9e7f6a4412d48fd99f6"
server_key_version: "encryptionkey@0"
I20260619 06:40:59.507790 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "74e482e505174fd696ac7f4c7c5c55b2"
format_stamp: "Formatted at 2026-06-19 06:40:59 on dist-test-slave-qfk4"
server_key: "dac8fb83efa701308b74c1fa99885e12"
server_key_iv: "1edf7a918b1ce9e7f6a4412d48fd99f6"
server_key_version: "encryptionkey@0"
I20260619 06:40:59.512607 17031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260619 06:40:59.515908 17045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:40:59.517297 17031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:40:59.517450 17031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "74e482e505174fd696ac7f4c7c5c55b2"
format_stamp: "Formatted at 2026-06-19 06:40:59 on dist-test-slave-qfk4"
server_key: "dac8fb83efa701308b74c1fa99885e12"
server_key_iv: "1edf7a918b1ce9e7f6a4412d48fd99f6"
server_key_version: "encryptionkey@0"
I20260619 06:40:59.517637 17031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:40:59.547928 17031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:40:59.548848 17031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:40:59.549110 17031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:40:59.566398 17031 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:38211
I20260619 06:40:59.566406 17096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:38211 every 8 connection(s)
I20260619 06:40:59.567734 17031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:40:59.569062 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 17031
I20260619 06:40:59.569269 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:40:59.570182 7755 external_mini_cluster.cc:1428] Setting key f0e2d1a9c58d2b1aa15eebd0b3a27438
I20260619 06:40:59.572747 17097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:40:59.584430 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:40:59.572389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.7.146.254:38211)
0619 06:40:59.573386 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0619 06:40:59.573397 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:40:59.573439 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:40:59.575004 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:40:59.575024 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:40:59.575510 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:40:59.575851 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:40:59.576484 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:59.577864 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:59.579710 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:40:59.580480 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:40:59.580654 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:40:59.581910 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:40:59.581918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:40:59.581935 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:40:59.582541 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:40:59.582623 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:40:59.582648 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:40:59.582747 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:40:59.583039 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:40:59.583415 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":362,"threads_started":1}
I20260619 06:40:59.585456 17097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2: Bootstrap starting.
I20260619 06:40:59.588670 17097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2: Neither blocks nor log segments found. Creating new log.
I20260619 06:40:59.589789 17097 log.cc:824] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:40:59.592433 17097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2: No bootstrap required, opened a new log
I20260619 06:40:59.599743 17097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e482e505174fd696ac7f4c7c5c55b2" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38211 } }
I20260619 06:40:59.600143 17097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:40:59.600248 17097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e482e505174fd696ac7f4c7c5c55b2, State: Initialized, Role: FOLLOWER
I20260619 06:40:59.600870 17097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e482e505174fd696ac7f4c7c5c55b2" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38211 } }
I20260619 06:40:59.601083 17097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:40:59.601207 17097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:40:59.601370 17097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:40:59.603564 17097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e482e505174fd696ac7f4c7c5c55b2" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38211 } }
I20260619 06:40:59.604112 17097 leader_election.cc:302] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74e482e505174fd696ac7f4c7c5c55b2; no voters:
I20260619 06:40:59.604759 17097 leader_election.cc:288] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:40:59.604925 17102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:40:59.605547 17102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [term 1 LEADER]: Becoming Leader. State: Replica: 74e482e505174fd696ac7f4c7c5c55b2, State: Running, Role: LEADER
I20260619 06:40:59.606105 17102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e482e505174fd696ac7f4c7c5c55b2" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38211 } }
I20260619 06:40:59.606693 17097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:40:59.608732 17104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74e482e505174fd696ac7f4c7c5c55b2. Latest consensus state: current_term: 1 leader_uuid: "74e482e505174fd696ac7f4c7c5c55b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e482e505174fd696ac7f4c7c5c55b2" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38211 } } }
I20260619 06:40:59.608888 17103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74e482e505174fd696ac7f4c7c5c55b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e482e505174fd696ac7f4c7c5c55b2" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38211 } } }
I20260619 06:40:59.609216 17103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:59.609124 17104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2 [sys.catalog]: This master's current role is: LEADER
I20260619 06:40:59.613942 17110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:40:59.618019 17110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:40:59.626447 17110 catalog_manager.cc:1347] Generated new cluster ID: b8a267ac9b464585b6d693fa0db0bc78
I20260619 06:40:59.626551 17110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:40:59.658628 17110 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:40:59.659849 17110 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:40:59.667542 17110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74e482e505174fd696ac7f4c7c5c55b2: Generated new TSK 0
I20260619 06:40:59.668308 17110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:40:59.685415 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38211
--builtin_ntp_servers=127.7.146.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 06:41:00.004240 17121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:00.004544 17121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:00.004686 17121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:00.012873 17121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:00.013001 17121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:00.013139 17121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:41:00.023310 17121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:00.024688 17121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:00.026516 17121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:00.040096 17127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:00.040205 17126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:00.040591 17129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:00.040937 17121 server_base.cc:1034] running on GCE node
I20260619 06:41:00.041563 17121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:00.042582 17121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:00.043820 17121 hybrid_clock.cc:648] HybridClock initialized: now 1781851260043756 us; error 58 us; skew 500 ppm
I20260619 06:41:00.044176 17121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:41:00.046550 17121 webserver.cc:460] Webserver started at http://127.7.146.193:45961/ using document root <none> and password file <none>
I20260619 06:41:00.047263 17121 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:00.047367 17121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:00.047674 17121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:41:00.050621 17121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "9668afd6344d423d91e1a50cb0ca8947"
format_stamp: "Formatted at 2026-06-19 06:41:00 on dist-test-slave-qfk4"
server_key: "70086068a6977c5e3c045ce8418eae0c"
server_key_iv: "2df487d49ceca2f3f5cc6845dcf2d188"
server_key_version: "encryptionkey@0"
I20260619 06:41:00.051420 17121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "9668afd6344d423d91e1a50cb0ca8947"
format_stamp: "Formatted at 2026-06-19 06:41:00 on dist-test-slave-qfk4"
server_key: "70086068a6977c5e3c045ce8418eae0c"
server_key_iv: "2df487d49ceca2f3f5cc6845dcf2d188"
server_key_version: "encryptionkey@0"
I20260619 06:41:00.056892 17121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260619 06:41:00.060711 17135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:00.062197 17121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:41:00.062369 17121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "9668afd6344d423d91e1a50cb0ca8947"
format_stamp: "Formatted at 2026-06-19 06:41:00 on dist-test-slave-qfk4"
server_key: "70086068a6977c5e3c045ce8418eae0c"
server_key_iv: "2df487d49ceca2f3f5cc6845dcf2d188"
server_key_version: "encryptionkey@0"
I20260619 06:41:00.062580 17121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:00.077346 17121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:00.078244 17121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:00.078524 17121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:00.079486 17121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:00.081032 17121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:00.081135 17121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:00.081246 17121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:00.081315 17121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:00.111476 17121 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:40065
I20260619 06:41:00.111528 17247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:40065 every 8 connection(s)
I20260619 06:41:00.112861 17121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:41:00.119016 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 17121
I20260619 06:41:00.119194 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:41:00.119794 7755 external_mini_cluster.cc:1428] Setting key 5a224a428cbd5674162e76c26ba48426
I20260619 06:41:00.122607 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38211
--builtin_ntp_servers=127.7.146.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:41:00.126219 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:00.115368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:54231 (local address 127.7.146.254:38211)
0619 06:41:00.115789 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:00.115795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:00.117090 (+ 1295us) server_negotiation.cc:407] Connection header received
0619 06:41:00.118954 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:00.118959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:00.119028 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:00.119139 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:00.120424 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:00.121330 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:00.123236 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:00.123851 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:00.123991 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:00.125373 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:00.125380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:00.125384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:00.125407 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:00.125451 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:00.125465 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:00.125565 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:00.125786 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 06:41:00.125884 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":197,"threads_started":1}
I20260619 06:41:00.127218 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:00.115970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38211 (local address 127.7.146.193:54231)
0619 06:41:00.116925 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:00.116963 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:00.118597 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:00.119429 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:00.119438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:00.119535 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:00.120236 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:00.120256 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:00.121491 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:00.121497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:00.123068 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:00.123083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:00.124062 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:00.124068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:00.124275 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:00.125009 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:41:00.125012 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:00.125061 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:41:00.125147 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:00.125577 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:00.125586 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:00.125594 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:41:00.125760 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 06:41:00.126130 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":270,"threads_started":1}
I20260619 06:41:00.128778 17248 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38211
I20260619 06:41:00.129076 17248 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:00.129851 17248 heartbeater.cc:511] Master 127.7.146.254:38211 requested a full tablet report, sending...
I20260619 06:41:00.132122 17062 ts_manager.cc:194] Registered new tserver with Master: 9668afd6344d423d91e1a50cb0ca8947 (127.7.146.193:40065)
I20260619 06:41:00.134081 17062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.193:54231
W20260619 06:41:00.449712 17252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:00.450004 17252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:00.450085 17252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:00.457269 17252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:00.457376 17252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:00.457477 17252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:41:00.466578 17252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:00.467832 17252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:00.469503 17252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:00.482251 17260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:00.482290 17258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:00.482307 17257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:00.482872 17252 server_base.cc:1034] running on GCE node
I20260619 06:41:00.483449 17252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:00.484373 17252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:00.485581 17252 hybrid_clock.cc:648] HybridClock initialized: now 1781851260485517 us; error 52 us; skew 500 ppm
I20260619 06:41:00.485925 17252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:41:00.488092 17252 webserver.cc:460] Webserver started at http://127.7.146.194:36503/ using document root <none> and password file <none>
I20260619 06:41:00.488830 17252 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:00.488929 17252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:00.489238 17252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:41:00.492002 17252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "13df205cb91e45409c27341cd1e3e4c3"
format_stamp: "Formatted at 2026-06-19 06:41:00 on dist-test-slave-qfk4"
server_key: "6bd27a07c24d9dffd429f22c3baead79"
server_key_iv: "46e0c6e769b0d99f621294bae08165cc"
server_key_version: "encryptionkey@0"
I20260619 06:41:00.492815 17252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "13df205cb91e45409c27341cd1e3e4c3"
format_stamp: "Formatted at 2026-06-19 06:41:00 on dist-test-slave-qfk4"
server_key: "6bd27a07c24d9dffd429f22c3baead79"
server_key_iv: "46e0c6e769b0d99f621294bae08165cc"
server_key_version: "encryptionkey@0"
I20260619 06:41:00.497954 17252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:41:00.501560 17266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:00.503052 17252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:41:00.503209 17252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "13df205cb91e45409c27341cd1e3e4c3"
format_stamp: "Formatted at 2026-06-19 06:41:00 on dist-test-slave-qfk4"
server_key: "6bd27a07c24d9dffd429f22c3baead79"
server_key_iv: "46e0c6e769b0d99f621294bae08165cc"
server_key_version: "encryptionkey@0"
I20260619 06:41:00.503415 17252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:00.522045 17252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:00.522893 17252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:00.523168 17252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:00.524113 17252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:00.525578 17252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:00.525681 17252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:00.525792 17252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:00.525862 17252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:00.556461 17252 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:45553
I20260619 06:41:00.556514 17378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:45553 every 8 connection(s)
I20260619 06:41:00.557819 17252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:41:00.567780 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 17252
I20260619 06:41:00.567960 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:41:00.568547 7755 external_mini_cluster.cc:1428] Setting key 41f8502de867b7d5fe03d80611848753
I20260619 06:41:00.571460 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:00.560333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:35573 (local address 127.7.146.254:38211)
0619 06:41:00.560496 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:00.560500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:00.562255 (+ 1755us) server_negotiation.cc:407] Connection header received
0619 06:41:00.564201 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:00.564205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:00.564288 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:00.564390 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:00.565762 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:00.566902 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:00.568598 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:00.569058 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:00.569170 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:00.570644 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:00.570649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:00.570652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:00.570671 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:00.570708 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:00.570720 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:00.570764 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:00.571034 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:41:00.571128 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:41:00.571714 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38211
--builtin_ntp_servers=127.7.146.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:41:00.572600 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:00.561016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38211 (local address 127.7.146.194:35573)
0619 06:41:00.562056 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:00.562095 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:00.563871 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:00.564716 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:00.564725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:00.564864 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:00.565484 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:00.565503 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:00.567038 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:00.567043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:00.568428 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:00.568445 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:00.569274 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:00.569282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:00.569493 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:00.570277 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:41:00.570280 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:00.570335 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:41:00.570439 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:00.570844 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:00.570851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:00.570856 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:41:00.571005 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 06:41:00.571413 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":306,"threads_started":1}
I20260619 06:41:00.577922 17379 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38211
I20260619 06:41:00.578240 17379 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:00.579093 17379 heartbeater.cc:511] Master 127.7.146.254:38211 requested a full tablet report, sending...
I20260619 06:41:00.580660 17062 ts_manager.cc:194] Registered new tserver with Master: 13df205cb91e45409c27341cd1e3e4c3 (127.7.146.194:45553)
I20260619 06:41:00.581506 17062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.194:35573
W20260619 06:41:00.888340 17382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:00.888664 17382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:00.888800 17382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:00.896139 17382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:00.896245 17382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:00.896376 17382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:41:00.905614 17382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:00.906980 17382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:00.908872 17382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:00.921792 17387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:00.921666 17388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:00.921705 17390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:00.923076 17382 server_base.cc:1034] running on GCE node
I20260619 06:41:00.923743 17382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:00.924803 17382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:00.926031 17382 hybrid_clock.cc:648] HybridClock initialized: now 1781851260925975 us; error 49 us; skew 500 ppm
I20260619 06:41:00.926378 17382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:41:00.928794 17382 webserver.cc:460] Webserver started at http://127.7.146.195:35465/ using document root <none> and password file <none>
I20260619 06:41:00.929540 17382 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:00.929646 17382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:00.929973 17382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:41:00.932979 17382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "42902207cf134d1d99ab332b324a8bbf"
format_stamp: "Formatted at 2026-06-19 06:41:00 on dist-test-slave-qfk4"
server_key: "d559137c7cf14fc754f3a09d055bccc6"
server_key_iv: "01d2320ee50fbdefd9b55c10d74ba832"
server_key_version: "encryptionkey@0"
I20260619 06:41:00.933756 17382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "42902207cf134d1d99ab332b324a8bbf"
format_stamp: "Formatted at 2026-06-19 06:41:00 on dist-test-slave-qfk4"
server_key: "d559137c7cf14fc754f3a09d055bccc6"
server_key_iv: "01d2320ee50fbdefd9b55c10d74ba832"
server_key_version: "encryptionkey@0"
I20260619 06:41:00.938953 17382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:41:00.942466 17396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:00.943848 17382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:41:00.943992 17382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "42902207cf134d1d99ab332b324a8bbf"
format_stamp: "Formatted at 2026-06-19 06:41:00 on dist-test-slave-qfk4"
server_key: "d559137c7cf14fc754f3a09d055bccc6"
server_key_iv: "01d2320ee50fbdefd9b55c10d74ba832"
server_key_version: "encryptionkey@0"
I20260619 06:41:00.944183 17382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:00.975729 17382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:00.976644 17382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:00.976933 17382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:00.977882 17382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:00.979282 17382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:00.979377 17382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:00.979481 17382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:00.979547 17382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:01.009586 17382 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:37769
I20260619 06:41:01.009647 17508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:37769 every 8 connection(s)
I20260619 06:41:01.010960 17382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:41:01.016273 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 17382
I20260619 06:41:01.016443 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:41:01.016978 7755 external_mini_cluster.cc:1428] Setting key ff73395656db65ed7ed98ab72f71e6ec
I20260619 06:41:01.024098 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:01.013535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:48531 (local address 127.7.146.254:38211)
0619 06:41:01.013710 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:01.013717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:01.015141 (+ 1424us) server_negotiation.cc:407] Connection header received
0619 06:41:01.017314 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:01.017319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:01.017390 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:01.017481 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:01.018830 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:01.019890 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:01.021630 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:01.022058 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:01.022144 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:01.023406 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:01.023410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:01.023413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:01.023431 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:01.023460 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:01.023472 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:01.023535 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:01.023792 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 06:41:01.023870 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:41:01.025175 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:01.014076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38211 (local address 127.7.146.195:48531)
0619 06:41:01.014990 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:01.015027 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:01.017002 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:01.017684 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:01.017704 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:01.017822 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:01.018632 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:01.018660 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:01.020045 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:01.020050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:01.021502 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:01.021514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:01.022260 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:01.022267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:01.022472 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:01.023109 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:41:01.023111 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:01.023155 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:41:01.023237 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:01.023551 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:01.023559 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:01.023570 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:41:01.023777 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 06:41:01.024084 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":271,"threads_started":1}
I20260619 06:41:01.026693 17509 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38211
I20260619 06:41:01.026968 17509 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:01.027704 17509 heartbeater.cc:511] Master 127.7.146.254:38211 requested a full tablet report, sending...
I20260619 06:41:01.029117 17062 ts_manager.cc:194] Registered new tserver with Master: 42902207cf134d1d99ab332b324a8bbf (127.7.146.195:37769)
I20260619 06:41:01.029891 17062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.195:48531
I20260619 06:41:01.032287 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:41:01.038070 17250 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:41:01.036682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.7.146.254:38211)
0619 06:41:01.036934 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:01.036940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:01.037167 (+ 227us) server_negotiation.cc:407] Connection header received
0619 06:41:01.037339 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:01.037342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:01.037413 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:01.037491 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:01.037848 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51904: BlockingRecv error: recv got EOF from 127.0.0.1:51904 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:41:01.038604 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 17121
I20260619 06:41:01.046162 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:01.039468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.7.146.193:40065)
0619 06:41:01.039959 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:01.039966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:01.039991 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:41:01.040044 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:01.040049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:01.040284 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:01.040423 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:01.040867 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:01.041831 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:01.043072 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:01.043705 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:01.043829 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:01.044900 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:01.044909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:01.044922 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:01.045495 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:01.045545 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:01.045565 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:01.045663 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:01.045775 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 06:41:01.045884 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":242,"threads_started":1}
I20260619 06:41:01.047386 17243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:01.127957 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 17121
I20260619 06:41:01.140697 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 17252
I20260619 06:41:01.148494 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:01.141515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.7.146.194:45553)
0619 06:41:01.142071 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:01.142086 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:01.142112 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:41:01.142186 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:01.142193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:01.142407 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:01.142538 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:01.142898 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:01.143881 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:01.145232 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:01.145813 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:01.145910 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:01.146901 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:01.146911 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:01.146925 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:01.147605 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:01.147671 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:01.147729 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:01.147829 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:01.147975 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 06:41:01.148166 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":290,"threads_started":1}
I20260619 06:41:01.149860 17374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:01.230189 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 17252
I20260619 06:41:01.242544 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 17382
I20260619 06:41:01.250391 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:01.243494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.7.146.195:37769)
0619 06:41:01.244102 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:01.244109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:01.244139 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:41:01.244200 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:01.244206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:01.244422 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:01.244592 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:01.245055 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:01.246072 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:01.247280 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:01.247866 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:01.247974 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:01.249063 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:01.249072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:01.249085 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:01.249646 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:01.249760 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:01.249782 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:01.249877 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:01.249967 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 06:41:01.250086 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":314,"threads_started":1}
I20260619 06:41:01.251602 17504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:01.334664 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 17382
I20260619 06:41:01.346961 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 17031
I20260619 06:41:01.348201 17092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:01.449112 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 17031
2026-06-19T06:41:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2364 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T06:41:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:41:01Z Disabled control of system clock
I20260619 06:41:01.513211 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:45841
--webserver_interface=127.7.146.254
--webserver_port=0
--builtin_ntp_servers=127.7.146.212:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:45841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:41:01.826555 17528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:01.826839 17528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:01.826974 17528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:01.834321 17528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:41:01.834438 17528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:01.834499 17528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:01.834581 17528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:41:01.834649 17528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:41:01.843923 17528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:45841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:45841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:01.845224 17528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:01.846962 17528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:01.858553 17533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:01.858531 17536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:01.858474 17534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:01.859256 17528 server_base.cc:1034] running on GCE node
I20260619 06:41:01.860086 17528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:01.861574 17528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:01.862783 17528 hybrid_clock.cc:648] HybridClock initialized: now 1781851261862741 us; error 33 us; skew 500 ppm
I20260619 06:41:01.863121 17528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:41:01.865386 17528 webserver.cc:460] Webserver started at http://127.7.146.254:42879/ using document root <none> and password file <none>
I20260619 06:41:01.866083 17528 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:01.866166 17528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:01.866463 17528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:41:01.869256 17528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "682069922a334279962067b14ab58033"
format_stamp: "Formatted at 2026-06-19 06:41:01 on dist-test-slave-qfk4"
server_key: "84234437deaf5377854102f60f5cac7b"
server_key_iv: "c80508832a67e43cd5b47e34bcf8b46d"
server_key_version: "encryptionkey@0"
I20260619 06:41:01.870028 17528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "682069922a334279962067b14ab58033"
format_stamp: "Formatted at 2026-06-19 06:41:01 on dist-test-slave-qfk4"
server_key: "84234437deaf5377854102f60f5cac7b"
server_key_iv: "c80508832a67e43cd5b47e34bcf8b46d"
server_key_version: "encryptionkey@0"
I20260619 06:41:01.875041 17528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:41:01.878502 17542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:01.880013 17528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:41:01.880187 17528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "682069922a334279962067b14ab58033"
format_stamp: "Formatted at 2026-06-19 06:41:01 on dist-test-slave-qfk4"
server_key: "84234437deaf5377854102f60f5cac7b"
server_key_iv: "c80508832a67e43cd5b47e34bcf8b46d"
server_key_version: "encryptionkey@0"
I20260619 06:41:01.880420 17528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:01.886494 17528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:01.887295 17528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:01.887562 17528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:01.905036 17528 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:45841
I20260619 06:41:01.905071 17593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:45841 every 8 connection(s)
I20260619 06:41:01.906404 17528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:41:01.910104 17594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:41:01.912400 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 17528
I20260619 06:41:01.912638 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:41:01.913260 7755 external_mini_cluster.cc:1428] Setting key ae096e1df485795daf6b28dc25768651
I20260619 06:41:01.921252 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:01.915198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.7.146.254:45841)
0619 06:41:01.916167 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:01.916178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:01.916218 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:41:01.917830 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:01.917851 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:01.918295 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:01.918671 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:01.919663 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:01.919669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:01.919681 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:01.919695 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:01.919754 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:01.919772 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:01.919961 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:01.920089 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 06:41:01.920330 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":320,"threads_started":1}
I20260619 06:41:01.921299 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033: Bootstrap starting.
I20260619 06:41:01.924471 17594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033: Neither blocks nor log segments found. Creating new log.
I20260619 06:41:01.925411 17594 log.cc:824] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033: Log is configured to *not* fsync() on all Append() calls
I20260619 06:41:01.927891 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033: No bootstrap required, opened a new log
I20260619 06:41:01.935081 17594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682069922a334279962067b14ab58033" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45841 } }
I20260619 06:41:01.935436 17594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:41:01.935523 17594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682069922a334279962067b14ab58033, State: Initialized, Role: FOLLOWER
I20260619 06:41:01.936074 17594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682069922a334279962067b14ab58033" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45841 } }
I20260619 06:41:01.936283 17594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:41:01.936400 17594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:41:01.936544 17594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:41:01.938622 17594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682069922a334279962067b14ab58033" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45841 } }
I20260619 06:41:01.939168 17594 leader_election.cc:302] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 682069922a334279962067b14ab58033; no voters:
I20260619 06:41:01.939805 17594 leader_election.cc:288] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:41:01.939929 17599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:41:01.940548 17599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [term 1 LEADER]: Becoming Leader. State: Replica: 682069922a334279962067b14ab58033, State: Running, Role: LEADER
I20260619 06:41:01.941102 17599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682069922a334279962067b14ab58033" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45841 } }
I20260619 06:41:01.941821 17594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:41:01.943732 17601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 682069922a334279962067b14ab58033. Latest consensus state: current_term: 1 leader_uuid: "682069922a334279962067b14ab58033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682069922a334279962067b14ab58033" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45841 } } }
I20260619 06:41:01.945181 17601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [sys.catalog]: This master's current role is: LEADER
I20260619 06:41:01.944505 17600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "682069922a334279962067b14ab58033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682069922a334279962067b14ab58033" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45841 } } }
I20260619 06:41:01.945742 17600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033 [sys.catalog]: This master's current role is: LEADER
I20260619 06:41:01.946141 17609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:41:01.951440 17609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:41:01.960523 17609 catalog_manager.cc:1347] Generated new cluster ID: 9068ad2bb0b44d0ca5927993db0d857e
I20260619 06:41:01.960678 17609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:41:01.981889 17609 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:41:01.982297 17609 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:41:01.987601 17609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 682069922a334279962067b14ab58033: Generated new TSK 0
I20260619 06:41:01.988459 17609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:41:02.000293 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:45841
--builtin_ntp_servers=127.7.146.212:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:41:02.353159 17618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:02.353434 17618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:02.353521 17618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:02.360878 17618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:02.360986 17618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:02.361094 17618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:41:02.370376 17618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:45841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:02.371831 17618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:02.373674 17618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:02.386793 17624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:02.387123 17623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:02.387307 17626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:02.387655 17618 server_base.cc:1034] running on GCE node
I20260619 06:41:02.388303 17618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:02.389331 17618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:02.390576 17618 hybrid_clock.cc:648] HybridClock initialized: now 1781851262390493 us; error 52 us; skew 500 ppm
I20260619 06:41:02.390920 17618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:41:02.393236 17618 webserver.cc:460] Webserver started at http://127.7.146.193:37755/ using document root <none> and password file <none>
I20260619 06:41:02.393939 17618 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:02.394039 17618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:02.394345 17618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:41:02.397482 17618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "4c340209fd2c4addb5e1b71945a9b886"
format_stamp: "Formatted at 2026-06-19 06:41:02 on dist-test-slave-qfk4"
server_key: "ab419072b01db1a69be5af43fc4e5997"
server_key_iv: "98f6d638e91e0f9b3c3288fcb143d460"
server_key_version: "encryptionkey@0"
I20260619 06:41:02.398307 17618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "4c340209fd2c4addb5e1b71945a9b886"
format_stamp: "Formatted at 2026-06-19 06:41:02 on dist-test-slave-qfk4"
server_key: "ab419072b01db1a69be5af43fc4e5997"
server_key_iv: "98f6d638e91e0f9b3c3288fcb143d460"
server_key_version: "encryptionkey@0"
I20260619 06:41:02.403735 17618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:41:02.407368 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:02.408913 17618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:41:02.409076 17618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "4c340209fd2c4addb5e1b71945a9b886"
format_stamp: "Formatted at 2026-06-19 06:41:02 on dist-test-slave-qfk4"
server_key: "ab419072b01db1a69be5af43fc4e5997"
server_key_iv: "98f6d638e91e0f9b3c3288fcb143d460"
server_key_version: "encryptionkey@0"
I20260619 06:41:02.409278 17618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:02.414829 17618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:02.415611 17618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:02.415872 17618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:02.416817 17618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:02.418207 17618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:02.418306 17618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:02.418414 17618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:02.418483 17618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:02.448479 17618 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:44933
I20260619 06:41:02.448617 17744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:44933 every 8 connection(s)
I20260619 06:41:02.449824 17618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:41:02.458103 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 17618
I20260619 06:41:02.458302 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:41:02.458402 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:02.452525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:54445 (local address 127.7.146.254:45841)
0619 06:41:02.453019 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:02.453024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:02.454273 (+ 1249us) server_negotiation.cc:407] Connection header received
0619 06:41:02.456077 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:02.456083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:02.456154 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:02.456265 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:02.457582 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:02.457588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:02.457594 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:02.457622 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:02.457666 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:02.457680 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:02.457804 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:02.457984 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 06:41:02.458053 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":242,"threads_started":1}
I20260619 06:41:02.458974 7755 external_mini_cluster.cc:1428] Setting key 816bba589a379b8cb1cf8569d66473bd
I20260619 06:41:02.459367 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:02.453048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:45841 (local address 127.7.146.193:54445)
0619 06:41:02.454090 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:02.454146 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:02.455786 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:02.456520 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:02.456529 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:02.456667 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:02.457281 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:41:02.457283 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:02.457322 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:41:02.457412 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:02.457770 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:02.457782 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:02.457790 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:41:02.458059 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 06:41:02.458330 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":253,"threads_started":1}
I20260619 06:41:02.460901 17745 heartbeater.cc:347] Connected to a master server at 127.7.146.254:45841
I20260619 06:41:02.461202 17745 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:02.461932 17745 heartbeater.cc:511] Master 127.7.146.254:45841 requested a full tablet report, sending...
I20260619 06:41:02.462100 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:45841
--builtin_ntp_servers=127.7.146.212:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 06:41:02.464110 17559 ts_manager.cc:194] Registered new tserver with Master: 4c340209fd2c4addb5e1b71945a9b886 (127.7.146.193:44933)
W20260619 06:41:02.771697 17749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:02.771978 17749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:02.772063 17749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:02.779228 17749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:02.779335 17749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:02.779444 17749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:41:02.788771 17749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:45841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:02.790010 17749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:02.791646 17749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:02.804196 17754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:02.804164 17755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:02.804231 17757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:02.805073 17749 server_base.cc:1034] running on GCE node
I20260619 06:41:02.805824 17749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:02.806850 17749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:02.808046 17749 hybrid_clock.cc:648] HybridClock initialized: now 1781851262807991 us; error 48 us; skew 500 ppm
I20260619 06:41:02.808350 17749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:41:02.810482 17749 webserver.cc:460] Webserver started at http://127.7.146.194:35855/ using document root <none> and password file <none>
I20260619 06:41:02.811136 17749 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:02.811231 17749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:02.811537 17749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:41:02.814452 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "f231f3f8c3c447b1911ef2fa43c8d519"
format_stamp: "Formatted at 2026-06-19 06:41:02 on dist-test-slave-qfk4"
server_key: "a73734318ffe93b104fc80437a72c642"
server_key_iv: "24aa99bad8d4554ea371598d1afd5a70"
server_key_version: "encryptionkey@0"
I20260619 06:41:02.815169 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "f231f3f8c3c447b1911ef2fa43c8d519"
format_stamp: "Formatted at 2026-06-19 06:41:02 on dist-test-slave-qfk4"
server_key: "a73734318ffe93b104fc80437a72c642"
server_key_iv: "24aa99bad8d4554ea371598d1afd5a70"
server_key_version: "encryptionkey@0"
I20260619 06:41:02.819866 17749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 06:41:02.823249 17763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:02.824736 17749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:41:02.824878 17749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "f231f3f8c3c447b1911ef2fa43c8d519"
format_stamp: "Formatted at 2026-06-19 06:41:02 on dist-test-slave-qfk4"
server_key: "a73734318ffe93b104fc80437a72c642"
server_key_iv: "24aa99bad8d4554ea371598d1afd5a70"
server_key_version: "encryptionkey@0"
I20260619 06:41:02.825066 17749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:02.830366 17749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:02.831090 17749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:02.831336 17749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:02.832187 17749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:02.833547 17749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:02.833639 17749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:02.833743 17749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:02.833810 17749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:02.862392 17749 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:36017
I20260619 06:41:02.862439 17875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:36017 every 8 connection(s)
I20260619 06:41:02.863675 17749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:41:02.871488 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:02.866077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:52633 (local address 127.7.146.254:45841)
0619 06:41:02.866244 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:02.866249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:02.867668 (+ 1419us) server_negotiation.cc:407] Connection header received
0619 06:41:02.869390 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:02.869397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:02.869463 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:02.869550 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:02.870768 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:02.870774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:02.870778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:02.870798 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:02.870837 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:02.870848 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:02.870950 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:02.871114 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 06:41:02.871191 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:41:02.872354 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:02.866598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:45841 (local address 127.7.146.194:52633)
0619 06:41:02.867505 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:02.867540 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:02.869127 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:02.869795 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:02.869804 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:02.869899 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:02.870478 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:41:02.870481 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:02.870517 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:41:02.870602 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:02.870928 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:02.870937 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:02.870944 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:41:02.871187 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 06:41:02.871440 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":249,"threads_started":1}
I20260619 06:41:02.872879 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 17749
I20260619 06:41:02.873042 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:41:02.873442 7755 external_mini_cluster.cc:1428] Setting key 8d1d1e1ba5d4b99b2ed6aa695058ec68
I20260619 06:41:02.873699 17876 heartbeater.cc:347] Connected to a master server at 127.7.146.254:45841
I20260619 06:41:02.873999 17876 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:02.874532 17876 heartbeater.cc:511] Master 127.7.146.254:45841 requested a full tablet report, sending...
I20260619 06:41:02.875838 17559 ts_manager.cc:194] Registered new tserver with Master: f231f3f8c3c447b1911ef2fa43c8d519 (127.7.146.194:36017)
I20260619 06:41:02.876106 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:45841
--builtin_ntp_servers=127.7.146.212:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:41:03.193521 17879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:03.193823 17879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:03.193951 17879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:03.201285 17879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:03.201383 17879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:03.201534 17879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:41:03.210757 17879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:45841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:03.212041 17879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:03.214097 17879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:03.229441 17884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:03.229365 17887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:03.229357 17885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:03.230599 17879 server_base.cc:1034] running on GCE node
I20260619 06:41:03.231211 17879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:03.232131 17879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:03.233326 17879 hybrid_clock.cc:648] HybridClock initialized: now 1781851263233278 us; error 41 us; skew 500 ppm
I20260619 06:41:03.233623 17879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:41:03.235797 17879 webserver.cc:460] Webserver started at http://127.7.146.195:40567/ using document root <none> and password file <none>
I20260619 06:41:03.236459 17879 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:03.236557 17879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:03.236925 17879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:41:03.239650 17879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "d2b040f1210d43b7a15801dcb99f58e2"
format_stamp: "Formatted at 2026-06-19 06:41:03 on dist-test-slave-qfk4"
server_key: "cd38287ed97435ef7e5589264fcca27d"
server_key_iv: "0e7b2288d3150c020afbb6d0f553ba29"
server_key_version: "encryptionkey@0"
I20260619 06:41:03.240420 17879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "d2b040f1210d43b7a15801dcb99f58e2"
format_stamp: "Formatted at 2026-06-19 06:41:03 on dist-test-slave-qfk4"
server_key: "cd38287ed97435ef7e5589264fcca27d"
server_key_iv: "0e7b2288d3150c020afbb6d0f553ba29"
server_key_version: "encryptionkey@0"
I20260619 06:41:03.245645 17879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 06:41:03.249082 17893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:03.250433 17879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:41:03.250581 17879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "d2b040f1210d43b7a15801dcb99f58e2"
format_stamp: "Formatted at 2026-06-19 06:41:03 on dist-test-slave-qfk4"
server_key: "cd38287ed97435ef7e5589264fcca27d"
server_key_iv: "0e7b2288d3150c020afbb6d0f553ba29"
server_key_version: "encryptionkey@0"
I20260619 06:41:03.250789 17879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:03.256201 17879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:03.256985 17879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:03.257213 17879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:03.258077 17879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:03.259388 17879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:03.259476 17879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:03.259578 17879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:03.259645 17879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:03.290151 17879 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:33379
I20260619 06:41:03.290208 18005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:33379 every 8 connection(s)
I20260619 06:41:03.291406 17879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:41:03.296733 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 17879
I20260619 06:41:03.296921 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:41:03.297436 7755 external_mini_cluster.cc:1428] Setting key e7120254f35e1fc5547fa30c65e68857
I20260619 06:41:03.299942 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:03.293849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:41389 (local address 127.7.146.254:45841)
0619 06:41:03.294058 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:03.294062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:03.295494 (+ 1432us) server_negotiation.cc:407] Connection header received
0619 06:41:03.297487 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:03.297493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:03.297562 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:03.297657 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:03.299228 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:03.299234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:03.299238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:03.299257 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:03.299293 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:03.299305 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:03.299389 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:03.299601 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 06:41:03.299666 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 06:41:03.300863 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:03.294379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:45841 (local address 127.7.146.195:41389)
0619 06:41:03.295348 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:03.295389 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:03.297163 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:03.298013 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:03.298023 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:03.298132 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:03.298872 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:41:03.298876 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:03.298928 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:41:03.299042 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:03.299395 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:03.299405 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:03.299412 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:41:03.299638 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 06:41:03.299841 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":286,"threads_started":1}
I20260619 06:41:03.302114 18006 heartbeater.cc:347] Connected to a master server at 127.7.146.254:45841
I20260619 06:41:03.302358 18006 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:03.302876 18006 heartbeater.cc:511] Master 127.7.146.254:45841 requested a full tablet report, sending...
I20260619 06:41:03.304082 17559 ts_manager.cc:194] Registered new tserver with Master: d2b040f1210d43b7a15801dcb99f58e2 (127.7.146.195:33379)
I20260619 06:41:03.312530 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:41:03.319161 17747 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:41:03.317631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.7.146.254:45841)
0619 06:41:03.317864 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:03.317871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:03.318269 (+ 398us) server_negotiation.cc:407] Connection header received
0619 06:41:03.318445 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:03.318451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:03.318500 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:03.318579 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:03.318917 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40564: BlockingRecv error: recv got EOF from 127.0.0.1:40564 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 06:41:03.319568 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 17618
I20260619 06:41:03.323004 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:03.320425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.7.146.193:44933)
0619 06:41:03.320966 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:03.320976 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:03.321001 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:41:03.321054 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:03.321060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:03.321297 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:03.321430 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:03.322329 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:03.322336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:03.322348 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:03.322365 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:03.322404 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:03.322417 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:03.322531 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:03.322667 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 06:41:03.322765 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":246,"threads_started":1}
I20260619 06:41:03.324234 17740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:03.404167 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 17618
I20260619 06:41:03.416814 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 17749
I20260619 06:41:03.420137 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:03.417821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.7.146.194:36017)
0619 06:41:03.418325 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:03.418332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:03.418358 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:41:03.418417 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:03.418423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:03.418632 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:03.418751 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:03.419574 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:03.419581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:03.419593 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:03.419611 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:03.419653 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:03.419666 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:03.419764 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:03.419837 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 06:41:03.419914 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":252,"threads_started":1}
I20260619 06:41:03.421464 17871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:03.501684 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 17749
I20260619 06:41:03.514246 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 17879
I20260619 06:41:03.518163 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:03.515166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.7.146.195:33379)
0619 06:41:03.515812 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:03.515819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:03.515848 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:41:03.515909 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:03.515915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:03.516131 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:03.516300 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:03.517439 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:03.517445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:03.517457 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:41:03.517472 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:03.517511 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:41:03.517525 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:03.517687 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:03.517764 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 06:41:03.517879 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":314,"threads_started":1}
I20260619 06:41:03.519382 18001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:03.599654 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 17879
I20260619 06:41:03.611393 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 17528
I20260619 06:41:03.612532 17589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:03.710629 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 17528
2026-06-19T06:41:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2260 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:41:03 dist-test-slave-qfk4 krb5kdc[18024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:41:03 dist-test-slave-qfk4 krb5kdc[18024](info): set up 2 sockets
Jun 19 06:41:03 dist-test-slave-qfk4 krb5kdc[18024](info): commencing operation
krb5kdc: starting...
W20260619 06:41:04.725200 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:41:04 dist-test-slave-qfk4 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:41:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:41:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:41:04.942041 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38059
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:38059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:41:05.255335 18037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:05.255609 18037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:05.255715 18037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:05.262970 18037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:41:05.263073 18037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:05.263118 18037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:05.263171 18037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:41:05.263213 18037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:41:05.272919 18037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:38059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:05.274180 18037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:05.275425 18037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:05.286289 18042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:05.286353 18045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:05.286378 18043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:05.287276 18037 server_base.cc:1034] running on GCE node
I20260619 06:41:05.287969 18037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:05.289343 18037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:05.290609 18037 hybrid_clock.cc:648] HybridClock initialized: now 1781851265290562 us; error 29 us; skew 500 ppm
I20260619 06:41:05.290979 18037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:41:05 dist-test-slave-qfk4 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851265, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:41:05.294051 18037 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:41:05.295467 18037 webserver.cc:460] Webserver started at http://127.7.146.254:39429/ using document root <none> and password file <none>
I20260619 06:41:05.296109 18037 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:05.296207 18037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:05.296444 18037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:41:05.299367 18037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "ab4dd62cb8ea42b6907702dc400e8196"
format_stamp: "Formatted at 2026-06-19 06:41:05 on dist-test-slave-qfk4"
server_key: "b573558352b186b511aa7819d9ef9937"
server_key_iv: "13fe1f67d8514a88e565cba28190b172"
server_key_version: "encryptionkey@0"
I20260619 06:41:05.300071 18037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "ab4dd62cb8ea42b6907702dc400e8196"
format_stamp: "Formatted at 2026-06-19 06:41:05 on dist-test-slave-qfk4"
server_key: "b573558352b186b511aa7819d9ef9937"
server_key_iv: "13fe1f67d8514a88e565cba28190b172"
server_key_version: "encryptionkey@0"
I20260619 06:41:05.305083 18037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260619 06:41:05.308511 18052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:05.309931 18037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:41:05.310065 18037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "ab4dd62cb8ea42b6907702dc400e8196"
format_stamp: "Formatted at 2026-06-19 06:41:05 on dist-test-slave-qfk4"
server_key: "b573558352b186b511aa7819d9ef9937"
server_key_iv: "13fe1f67d8514a88e565cba28190b172"
server_key_version: "encryptionkey@0"
I20260619 06:41:05.310216 18037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:05.332669 18037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:05.336594 18037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:05.336833 18037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:05.353808 18037 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:38059
I20260619 06:41:05.353859 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:38059 every 8 connection(s)
I20260619 06:41:05.355083 18037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:41:05.358749 18104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:41:05.362732 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 18037
I20260619 06:41:05.362963 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:41:05.363700 7755 external_mini_cluster.cc:1428] Setting key 9f597fa9789bac9f3b805233f3c5b31d
I20260619 06:41:05.370041 18104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196: Bootstrap starting.
I20260619 06:41:05.373575 18104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196: Neither blocks nor log segments found. Creating new log.
I20260619 06:41:05.374668 18104 log.cc:824] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196: Log is configured to *not* fsync() on all Append() calls
I20260619 06:41:05.377916 18104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196: No bootstrap required, opened a new log
Jun 19 06:41:05 dist-test-slave-qfk4 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:41:05.385462 18104 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4dd62cb8ea42b6907702dc400e8196" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38059 } }
I20260619 06:41:05.385823 18104 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:41:05.385957 18104 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4dd62cb8ea42b6907702dc400e8196, State: Initialized, Role: FOLLOWER
I20260619 06:41:05.386514 18104 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4dd62cb8ea42b6907702dc400e8196" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38059 } }
I20260619 06:41:05.386688 18107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:05.365781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.7.146.254:38059)
0619 06:41:05.366494 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:05.366506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:05.366549 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:41:05.368509 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:05.368533 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:05.368964 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:05.369280 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:05.370962 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:05.372346 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:05.374092 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:05.374836 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:05.374984 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:05.379389 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:05.379406 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:05.379424 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:05.379462 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:05.382542 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:05.383211 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:05.383221 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:05.383227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:05.383390 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:05.383743 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:05.383747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:05.383749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:05.384297 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:05.384650 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:05.385022 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 06:41:05.385427 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":337,"threads_started":1}
I20260619 06:41:05.386754 18104 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:41:05.386888 18104 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:41:05.387075 18104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:41:05.389787 18104 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4dd62cb8ea42b6907702dc400e8196" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38059 } }
I20260619 06:41:05.390249 18104 leader_election.cc:302] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab4dd62cb8ea42b6907702dc400e8196; no voters:
I20260619 06:41:05.390782 18104 leader_election.cc:288] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:41:05.390921 18109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:41:05.391901 18109 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [term 1 LEADER]: Becoming Leader. State: Replica: ab4dd62cb8ea42b6907702dc400e8196, State: Running, Role: LEADER
I20260619 06:41:05.392527 18109 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4dd62cb8ea42b6907702dc400e8196" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38059 } }
I20260619 06:41:05.392877 18104 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:41:05.396289 18110 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab4dd62cb8ea42b6907702dc400e8196" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4dd62cb8ea42b6907702dc400e8196" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38059 } } }
I20260619 06:41:05.396325 18111 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab4dd62cb8ea42b6907702dc400e8196. Latest consensus state: current_term: 1 leader_uuid: "ab4dd62cb8ea42b6907702dc400e8196" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4dd62cb8ea42b6907702dc400e8196" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38059 } } }
I20260619 06:41:05.396634 18110 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [sys.catalog]: This master's current role is: LEADER
I20260619 06:41:05.396701 18111 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196 [sys.catalog]: This master's current role is: LEADER
I20260619 06:41:05.405813 18120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:41:05.410986 18120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:41:05.419427 18120 catalog_manager.cc:1347] Generated new cluster ID: 0593ddb104494b42be0d3032ee1894a9
I20260619 06:41:05.419543 18120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:41:05.431500 18120 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:41:05.433354 18120 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:41:05.449077 18120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196: Generated new TSK 0
I20260619 06:41:05.450407 18120 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:41:05.512398 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38059
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:41:05.833366 18130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:05.833667 18130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:05.833807 18130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:05.841074 18130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:05.841197 18130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:05.841346 18130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:41:05.851147 18130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:05.852422 18130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:05.853767 18130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:05.866466 18136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:05.866443 18138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:05.866703 18135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:05.867074 18130 server_base.cc:1034] running on GCE node
I20260619 06:41:05.867686 18130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:05.868870 18130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:05.870078 18130 hybrid_clock.cc:648] HybridClock initialized: now 1781851265870032 us; error 43 us; skew 500 ppm
I20260619 06:41:05.870399 18130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:41:05 dist-test-slave-qfk4 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851265, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:41:05.873436 18130 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:41:05.874890 18130 webserver.cc:460] Webserver started at http://127.7.146.193:36117/ using document root <none> and password file <none>
I20260619 06:41:05.875557 18130 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:05.875676 18130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:05.875958 18130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:41:05.878898 18130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "4408951f5cc643ae972810e7e3a7e793"
format_stamp: "Formatted at 2026-06-19 06:41:05 on dist-test-slave-qfk4"
server_key: "4eb94ff5cb6f81998a65945aa10749a3"
server_key_iv: "534912b85afd43285c8da2f44a55ebd5"
server_key_version: "encryptionkey@0"
I20260619 06:41:05.879657 18130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "4408951f5cc643ae972810e7e3a7e793"
format_stamp: "Formatted at 2026-06-19 06:41:05 on dist-test-slave-qfk4"
server_key: "4eb94ff5cb6f81998a65945aa10749a3"
server_key_iv: "534912b85afd43285c8da2f44a55ebd5"
server_key_version: "encryptionkey@0"
I20260619 06:41:05.884784 18130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:41:05.888481 18145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:05.890036 18130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:41:05.890190 18130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "4408951f5cc643ae972810e7e3a7e793"
format_stamp: "Formatted at 2026-06-19 06:41:05 on dist-test-slave-qfk4"
server_key: "4eb94ff5cb6f81998a65945aa10749a3"
server_key_iv: "534912b85afd43285c8da2f44a55ebd5"
server_key_version: "encryptionkey@0"
I20260619 06:41:05.890385 18130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:05.904932 18130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:05.908996 18130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:05.909257 18130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:05.910156 18130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:05.911444 18130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:05.911517 18130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:05.911633 18130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:05.911686 18130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:05.941128 18130 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:44945
I20260619 06:41:05.941211 18257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:44945 every 8 connection(s)
I20260619 06:41:05.942361 18130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:41:05.949527 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 18130
I20260619 06:41:05.949734 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:41:05.950340 7755 external_mini_cluster.cc:1428] Setting key 649365dfe145abb3a04fbe708b2d6389
Jun 19 06:41:05 dist-test-slave-qfk4 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851265, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:41:05.961726 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:05.944837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:50507 (local address 127.7.146.254:38059)
0619 06:41:05.945267 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:05.945272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:05.946245 (+ 973us) server_negotiation.cc:407] Connection header received
0619 06:41:05.947966 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:05.947971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:05.948045 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:05.948138 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:05.949880 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:05.950611 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:05.952401 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:05.952851 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:05.952963 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:05.956279 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:05.956284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:05.956287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:05.956318 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:05.958733 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:05.959416 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:05.959420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:05.959424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:05.959567 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:05.959982 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:05.959986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:05.959988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:05.960294 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:05.960442 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:05.961155 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 06:41:05.961373 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":198,"threads_started":1}
I20260619 06:41:05.962560 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:05.945436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38059 (local address 127.7.146.193:50507)
0619 06:41:05.946074 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:05.946111 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:05.947687 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:05.948360 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:05.948369 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:05.948969 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:05.949678 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:05.949700 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:05.950757 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:05.950761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:05.952267 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:05.952280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:05.953065 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:05.953073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:05.953311 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:05.953967 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:41:05.953991 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:05.956062 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:05.958872 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:05.958878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:05.958891 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:05.959305 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:05.959670 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:05.959673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:05.959675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:05.959889 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:05.960414 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:05.960419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:05.960809 (+ 390us) client_negotiation.cc:769] Sending connection context
0619 06:41:05.961046 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 06:41:05.961434 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":259,"threads_started":1}
I20260619 06:41:05.964027 18258 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38059
I20260619 06:41:05.964329 18258 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:05.965061 18258 heartbeater.cc:511] Master 127.7.146.254:38059 requested a full tablet report, sending...
I20260619 06:41:05.967294 18069 ts_manager.cc:194] Registered new tserver with Master: 4408951f5cc643ae972810e7e3a7e793 (127.7.146.193:44945)
I20260619 06:41:05.969343 18069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:50507
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:41:05.997835 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38059
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:41:06.312848 18264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:06.313181 18264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:06.313338 18264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:06.320536 18264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:06.320686 18264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:06.320891 18264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:41:06.331533 18264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:06.332854 18264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:06.334141 18264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:06.347419 18272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:06.347477 18269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:06.347457 18270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:06.348423 18264 server_base.cc:1034] running on GCE node
I20260619 06:41:06.349094 18264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:06.350096 18264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:06.351296 18264 hybrid_clock.cc:648] HybridClock initialized: now 1781851266351237 us; error 49 us; skew 500 ppm
I20260619 06:41:06.351619 18264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:41:06 dist-test-slave-qfk4 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851266, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:41:06.355115 18264 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:41:06.368345 18264 webserver.cc:460] Webserver started at http://127.7.146.194:37663/ using document root <none> and password file <none>
I20260619 06:41:06.369124 18264 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:06.369258 18264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:06.369546 18264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:41:06.372469 18264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "4d1ee36c72ed4be6a61edf843195e758"
format_stamp: "Formatted at 2026-06-19 06:41:06 on dist-test-slave-qfk4"
server_key: "553d0339158053f2165a6111da8aa8ec"
server_key_iv: "200224e10dda42013ca738aec07f667f"
server_key_version: "encryptionkey@0"
I20260619 06:41:06.373287 18264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "4d1ee36c72ed4be6a61edf843195e758"
format_stamp: "Formatted at 2026-06-19 06:41:06 on dist-test-slave-qfk4"
server_key: "553d0339158053f2165a6111da8aa8ec"
server_key_iv: "200224e10dda42013ca738aec07f667f"
server_key_version: "encryptionkey@0"
I20260619 06:41:06.378408 18264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:41:06.381906 18279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:06.383220 18264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:41:06.383373 18264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "4d1ee36c72ed4be6a61edf843195e758"
format_stamp: "Formatted at 2026-06-19 06:41:06 on dist-test-slave-qfk4"
server_key: "553d0339158053f2165a6111da8aa8ec"
server_key_iv: "200224e10dda42013ca738aec07f667f"
server_key_version: "encryptionkey@0"
I20260619 06:41:06.383550 18264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:06.407187 18264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:06.411057 18264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:06.411316 18264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:06.412261 18264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:06.413726 18264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:06.413816 18264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:06.413928 18264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:06.413988 18264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:06.443464 18264 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:40529
I20260619 06:41:06.443542 18391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:40529 every 8 connection(s)
I20260619 06:41:06.445012 18264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:41:06.448760 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 18264
I20260619 06:41:06.448951 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:41:06.449445 7755 external_mini_cluster.cc:1428] Setting key 7f1729133faa79d83c704b3bf0a082c6
Jun 19 06:41:06 dist-test-slave-qfk4 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851266, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:41:06.465829 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:06.447695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:53249 (local address 127.7.146.254:38059)
0619 06:41:06.447900 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:06.447905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:06.449214 (+ 1309us) server_negotiation.cc:407] Connection header received
0619 06:41:06.451442 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:06.451446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:06.451509 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:06.451606 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:06.453171 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:06.454150 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:06.455840 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:06.456379 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:06.456497 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:06.460177 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:06.460182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:06.460185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:06.460217 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:06.462666 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:06.463516 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:06.463521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:06.463524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:06.463674 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:06.464113 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:06.464116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:06.464118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:06.464377 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:06.464488 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:06.465373 (+ 885us) server_negotiation.cc:299] Negotiation successful
0619 06:41:06.465578 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 06:41:06.467270 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:06.448202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38059 (local address 127.7.146.194:53249)
0619 06:41:06.449047 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:06.449092 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:06.451141 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:06.451832 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:06.451841 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:06.452350 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:06.453003 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:06.453021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:06.454327 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:06.454332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:06.455697 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:06.455711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:06.456616 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:06.456624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:06.456897 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:06.457552 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:41:06.457574 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:06.459911 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:06.462804 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:06.462812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:06.462830 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:06.463364 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:06.463771 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:06.463776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:06.463779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:06.464017 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:06.464484 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:06.464491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:06.464980 (+ 489us) client_negotiation.cc:769] Sending connection context
0619 06:41:06.465273 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 06:41:06.465818 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":319,"threads_started":1}
I20260619 06:41:06.468735 18392 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38059
I20260619 06:41:06.469030 18392 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:06.469740 18392 heartbeater.cc:511] Master 127.7.146.254:38059 requested a full tablet report, sending...
I20260619 06:41:06.471179 18069 ts_manager.cc:194] Registered new tserver with Master: 4d1ee36c72ed4be6a61edf843195e758 (127.7.146.194:40529)
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
I20260619 06:41:06.471998 18069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:53249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:41:06.497939 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38059
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:41:06.809160 18397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:06.809461 18397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:06.809556 18397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:06.816903 18397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:06.817015 18397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:06.817128 18397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:41:06.826647 18397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:06.827890 18397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:06.829173 18397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:06.842180 18402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:06.842350 18405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:06.842353 18403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:06.843883 18397 server_base.cc:1034] running on GCE node
I20260619 06:41:06.844751 18397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:06.845860 18397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:06.847071 18397 hybrid_clock.cc:648] HybridClock initialized: now 1781851266847013 us; error 47 us; skew 500 ppm
I20260619 06:41:06.847419 18397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:41:06 dist-test-slave-qfk4 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851266, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:41:06.850523 18397 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:41:06.851961 18397 webserver.cc:460] Webserver started at http://127.7.146.195:43969/ using document root <none> and password file <none>
I20260619 06:41:06.852674 18397 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:06.852777 18397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:06.853024 18397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:41:06.855841 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1"
format_stamp: "Formatted at 2026-06-19 06:41:06 on dist-test-slave-qfk4"
server_key: "47d3b08bf6251ad9f0e7ce8b41b83fd9"
server_key_iv: "718a11b6272e78ed5666ebeec60e36b1"
server_key_version: "encryptionkey@0"
I20260619 06:41:06.856554 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1"
format_stamp: "Formatted at 2026-06-19 06:41:06 on dist-test-slave-qfk4"
server_key: "47d3b08bf6251ad9f0e7ce8b41b83fd9"
server_key_iv: "718a11b6272e78ed5666ebeec60e36b1"
server_key_version: "encryptionkey@0"
I20260619 06:41:06.861688 18397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:41:06.865324 18412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:06.866653 18397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:41:06.866792 18397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1"
format_stamp: "Formatted at 2026-06-19 06:41:06 on dist-test-slave-qfk4"
server_key: "47d3b08bf6251ad9f0e7ce8b41b83fd9"
server_key_iv: "718a11b6272e78ed5666ebeec60e36b1"
server_key_version: "encryptionkey@0"
I20260619 06:41:06.866982 18397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:06.885922 18397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:06.889976 18397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:06.890241 18397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:06.891177 18397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:06.892499 18397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:06.892596 18397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:06.892710 18397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:06.892755 18397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:06.922302 18397 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:45829
I20260619 06:41:06.922348 18524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:45829 every 8 connection(s)
I20260619 06:41:06.923525 18397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:41:06.928376 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 18397
I20260619 06:41:06.928591 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:41:06.929128 7755 external_mini_cluster.cc:1428] Setting key 6df99aa1dc0f30f3dacde4a16b9215f3
Jun 19 06:41:06 dist-test-slave-qfk4 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851266, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:41:06.942765 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:06.925905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:45369 (local address 127.7.146.254:38059)
0619 06:41:06.926079 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:06.926084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:06.927102 (+ 1018us) server_negotiation.cc:407] Connection header received
0619 06:41:06.929151 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:06.929157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:06.929261 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:06.929348 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:06.930982 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:06.931818 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:06.933421 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:06.933823 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:06.933917 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:06.937044 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:06.937049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:06.937052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:06.937084 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:06.939281 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:06.939909 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:06.939912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:06.939915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:06.940034 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:06.940443 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:06.940449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:06.940452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:06.940897 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:06.941071 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:06.941726 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 06:41:06.941960 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:41:06.943207 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:06.926299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38059 (local address 127.7.146.195:45369)
0619 06:41:06.926960 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:06.926996 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:06.928863 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:06.929618 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:06.929629 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:06.930135 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:06.930814 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:06.930833 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:06.931954 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:06.931958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:06.933291 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:06.933302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:06.934000 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:06.934004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:06.934189 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:06.934798 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:41:06.934823 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:06.936850 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:06.939396 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:06.939402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:06.939415 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:06.939798 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:06.940110 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:06.940113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:06.940115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:06.940282 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:06.941024 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:06.941030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:06.941348 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 06:41:06.941556 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 06:41:06.941929 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":245,"threads_started":1}
I20260619 06:41:06.944643 18525 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38059
I20260619 06:41:06.944918 18525 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:06.945719 18525 heartbeater.cc:511] Master 127.7.146.254:38059 requested a full tablet report, sending...
I20260619 06:41:06.947050 18068 ts_manager.cc:194] Registered new tserver with Master: 27ace7e56e4b4f43bb1e0d51c4ecebb1 (127.7.146.195:45829)
I20260619 06:41:06.947875 18068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:45369
I20260619 06:41:06.956983 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:41:06.972466 18258 heartbeater.cc:503] Master 127.7.146.254:38059 was elected leader, sending a full tablet report...
I20260619 06:41:06.973069 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:06.961181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.7.146.254:38059)
0619 06:41:06.961377 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:06.961384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:06.961804 (+ 420us) server_negotiation.cc:407] Connection header received
0619 06:41:06.961972 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:06.961977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:06.962042 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:06.962132 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:06.963403 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:06.964156 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:06.965407 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:06.965804 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:06.965893 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:06.968053 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:06.968058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:06.968061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:06.968093 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:06.970601 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:06.971219 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:06.971225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:06.971228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:06.971374 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:06.971682 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:06.971686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:06.971688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:06.972035 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:06.972237 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:06.972532 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 06:41:06.972759 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:41:06.976867 18069 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:41:06.983453 18069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:41:07.016426 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:06.995851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.7.146.193:44945)
0619 06:41:07.007991 (+ 12140us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:07.008003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:07.009208 (+ 1205us) server_negotiation.cc:407] Connection header received
0619 06:41:07.009494 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:07.009502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:07.009720 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:07.009892 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:41:07.010482 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.011488 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.014177 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.015348 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.015468 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.015719 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:07.015941 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 06:41:07.016103 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11921,"thread_start_us":12620,"threads_started":1}
I20260619 06:41:07.016527 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:06.995478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44945 (local address 127.0.0.1:33628)
0619 06:41:07.009053 (+ 13575us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:07.009095 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:07.009283 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:07.009921 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:07.009929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:07.009965 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:41:07.010313 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:07.010323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.011613 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.011617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:07.014031 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:07.014043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.015531 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.015537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:07.015724 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.015774 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 06:41:07.015886 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 06:41:07.016066 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13370,"thread_start_us":13055,"threads_started":1}
I20260619 06:41:07.020540 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:06.998908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51012 (local address 127.7.146.194:40529)
0619 06:41:07.012379 (+ 13471us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:07.012388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:07.013630 (+ 1242us) server_negotiation.cc:407] Connection header received
0619 06:41:07.013823 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:07.013831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:07.014070 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:07.014247 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:41:07.014785 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.016224 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.018737 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.019646 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.019753 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.019890 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:07.020181 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 06:41:07.020297 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13322,"thread_start_us":13214,"threads_started":1}
I20260619 06:41:07.020660 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:06.998405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:40529 (local address 127.0.0.1:51012)
0619 06:41:07.013489 (+ 15084us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:07.013519 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:07.013628 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:07.014274 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:07.014282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:07.014306 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:41:07.014625 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:07.014636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.016354 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.016359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:07.018569 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:07.018583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.019847 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.019854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:07.020011 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.020041 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:41:07.020133 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:41:07.020249 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14976,"mutex_wait_us":20,"thread_start_us":14158,"threads_started":1}
I20260619 06:41:07.020558 18193 tablet_service.cc:1467] Processing CreateTablet for tablet c18ba4d85a6f44a6b9852d54b734bd3c (DEFAULT_TABLE table=test-table [id=92d4f9092dee49e69a5a1c4487915992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:41:07.021951 18193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18ba4d85a6f44a6b9852d54b734bd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:41:07.027724 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:06.995674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:45829 (local address 127.0.0.1:35064)
0619 06:41:07.012856 (+ 17182us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:07.012878 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:07.013019 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:07.013718 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:07.013723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:07.013745 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:41:07.014220 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:07.014232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.016312 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.016317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:07.025390 (+ 9073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:07.025406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.026904 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.026911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:07.027071 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.027105 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:41:07.027196 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:41:07.027330 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17018,"mutex_wait_us":149,"thread_start_us":17592,"threads_started":1}
I20260619 06:41:07.027753 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:06.996412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.7.146.195:45829)
0619 06:41:07.008466 (+ 12054us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:07.008473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:07.013003 (+ 4530us) server_negotiation.cc:407] Connection header received
0619 06:41:07.013196 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:07.013204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:07.013484 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:07.013688 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:41:07.014479 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.016142 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.025589 (+ 9447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.026707 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.026833 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.027064 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:07.027247 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 06:41:07.027399 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11907,"thread_start_us":11762,"threads_started":1}
I20260619 06:41:07.028162 18327 tablet_service.cc:1467] Processing CreateTablet for tablet c18ba4d85a6f44a6b9852d54b734bd3c (DEFAULT_TABLE table=test-table [id=92d4f9092dee49e69a5a1c4487915992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:41:07.029656 18327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18ba4d85a6f44a6b9852d54b734bd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:41:07.030840 18460 tablet_service.cc:1467] Processing CreateTablet for tablet c18ba4d85a6f44a6b9852d54b734bd3c (DEFAULT_TABLE table=test-table [id=92d4f9092dee49e69a5a1c4487915992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:41:07.031921 18460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18ba4d85a6f44a6b9852d54b734bd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:41:07.037698 18543 tablet_bootstrap.cc:492] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793: Bootstrap starting.
I20260619 06:41:07.041277 18543 tablet_bootstrap.cc:654] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793: Neither blocks nor log segments found. Creating new log.
I20260619 06:41:07.042513 18543 log.cc:824] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793: Log is configured to *not* fsync() on all Append() calls
I20260619 06:41:07.042620 18544 tablet_bootstrap.cc:492] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758: Bootstrap starting.
I20260619 06:41:07.045933 18544 tablet_bootstrap.cc:654] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758: Neither blocks nor log segments found. Creating new log.
I20260619 06:41:07.046223 18546 tablet_bootstrap.cc:492] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1: Bootstrap starting.
I20260619 06:41:07.047255 18544 log.cc:824] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758: Log is configured to *not* fsync() on all Append() calls
I20260619 06:41:07.049419 18543 tablet_bootstrap.cc:492] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793: No bootstrap required, opened a new log
I20260619 06:41:07.049702 18543 ts_tablet_manager.cc:1397] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.011s
I20260619 06:41:07.050037 18546 tablet_bootstrap.cc:654] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1: Neither blocks nor log segments found. Creating new log.
I20260619 06:41:07.051188 18546 log.cc:824] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:41:07.052033 18544 tablet_bootstrap.cc:492] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758: No bootstrap required, opened a new log
I20260619 06:41:07.052278 18544 ts_tablet_manager.cc:1397] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260619 06:41:07.054091 18546 tablet_bootstrap.cc:492] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1: No bootstrap required, opened a new log
I20260619 06:41:07.054349 18546 ts_tablet_manager.cc:1397] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 06:41:07.057051 18543 raft_consensus.cc:348] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408951f5cc643ae972810e7e3a7e793" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44945 } } peers { permanent_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45829 } } peers { permanent_uuid: "4d1ee36c72ed4be6a61edf843195e758" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40529 } }
I20260619 06:41:07.057552 18543 raft_consensus.cc:374] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:41:07.057650 18543 raft_consensus.cc:729] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4408951f5cc643ae972810e7e3a7e793, State: Initialized, Role: FOLLOWER
I20260619 06:41:07.058251 18543 consensus_queue.cc:260] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408951f5cc643ae972810e7e3a7e793" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44945 } } peers { permanent_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45829 } } peers { permanent_uuid: "4d1ee36c72ed4be6a61edf843195e758" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40529 } }
I20260619 06:41:07.060077 18543 ts_tablet_manager.cc:1428] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
I20260619 06:41:07.061547 18544 raft_consensus.cc:348] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408951f5cc643ae972810e7e3a7e793" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44945 } } peers { permanent_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45829 } } peers { permanent_uuid: "4d1ee36c72ed4be6a61edf843195e758" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40529 } }
I20260619 06:41:07.061885 18544 raft_consensus.cc:374] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:41:07.061982 18544 raft_consensus.cc:729] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1ee36c72ed4be6a61edf843195e758, State: Initialized, Role: FOLLOWER
I20260619 06:41:07.062516 18544 consensus_queue.cc:260] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408951f5cc643ae972810e7e3a7e793" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44945 } } peers { permanent_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45829 } } peers { permanent_uuid: "4d1ee36c72ed4be6a61edf843195e758" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40529 } }
I20260619 06:41:07.064357 18392 heartbeater.cc:503] Master 127.7.146.254:38059 was elected leader, sending a full tablet report...
I20260619 06:41:07.064446 18544 ts_tablet_manager.cc:1428] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758: Time spent starting tablet: real 0.012s user 0.007s sys 0.006s
I20260619 06:41:07.064888 18546 raft_consensus.cc:348] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408951f5cc643ae972810e7e3a7e793" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44945 } } peers { permanent_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45829 } } peers { permanent_uuid: "4d1ee36c72ed4be6a61edf843195e758" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40529 } }
I20260619 06:41:07.065372 18546 raft_consensus.cc:374] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:41:07.065471 18546 raft_consensus.cc:729] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ace7e56e4b4f43bb1e0d51c4ecebb1, State: Initialized, Role: FOLLOWER
I20260619 06:41:07.066128 18546 consensus_queue.cc:260] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408951f5cc643ae972810e7e3a7e793" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44945 } } peers { permanent_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45829 } } peers { permanent_uuid: "4d1ee36c72ed4be6a61edf843195e758" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40529 } }
I20260619 06:41:07.067160 18525 heartbeater.cc:503] Master 127.7.146.254:38059 was elected leader, sending a full tablet report...
I20260619 06:41:07.067675 18546 ts_tablet_manager.cc:1428] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20260619 06:41:07.075445 18551 raft_consensus.cc:482] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:41:07.075688 18551 raft_consensus.cc:504] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408951f5cc643ae972810e7e3a7e793" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44945 } } peers { permanent_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45829 } } peers { permanent_uuid: "4d1ee36c72ed4be6a61edf843195e758" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40529 } }
I20260619 06:41:07.077406 18551 leader_election.cc:288] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4408951f5cc643ae972810e7e3a7e793 (127.7.146.193:44945), 4d1ee36c72ed4be6a61edf843195e758 (127.7.146.194:40529)
I20260619 06:41:07.083161 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:07.077686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:43035 (local address 127.7.146.193:44945)
0619 06:41:07.077908 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:07.077916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:07.077985 (+ 69us) server_negotiation.cc:407] Connection header received
0619 06:41:07.078335 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:07.078341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:07.078407 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:07.078500 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:41:07.078912 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.079634 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.081416 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.082245 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.082415 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.082499 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:07.082759 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 06:41:07.082890 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:41:07.083231 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:07.077594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44945 (local address 127.7.146.195:43035)
0619 06:41:07.077909 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:07.077926 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:07.078054 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:07.078539 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:07.078545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:07.078579 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:41:07.078774 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:07.078784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.079783 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.079790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:07.081259 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:07.081271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.082400 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.082406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:07.082608 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.082633 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:41:07.082736 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:41:07.082877 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1}
I20260619 06:41:07.083379 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:07.078083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:54101 (local address 127.7.146.194:40529)
0619 06:41:07.078246 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:07.078251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:07.078673 (+ 422us) server_negotiation.cc:407] Connection header received
0619 06:41:07.078824 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:07.078828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:07.078883 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:07.078951 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:41:07.079424 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.080117 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.081812 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.082568 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.082706 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.082784 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:07.083032 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 06:41:07.083156 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 06:41:07.083721 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:07.077745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:40529 (local address 127.7.146.195:54101)
0619 06:41:07.078580 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:07.078594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:07.078705 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:07.078972 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:07.078975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:07.078997 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:41:07.079285 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:07.079294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.080264 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.080268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:07.081660 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:07.081671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.082745 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.082751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:07.082893 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.082915 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:41:07.082990 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 06:41:07.083110 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":78,"thread_start_us":500,"threads_started":1}
I20260619 06:41:07.084414 18213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18ba4d85a6f44a6b9852d54b734bd3c" candidate_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4408951f5cc643ae972810e7e3a7e793" is_pre_election: true
I20260619 06:41:07.084681 18347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18ba4d85a6f44a6b9852d54b734bd3c" candidate_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1ee36c72ed4be6a61edf843195e758" is_pre_election: true
I20260619 06:41:07.084928 18213 raft_consensus.cc:2454] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ace7e56e4b4f43bb1e0d51c4ecebb1 in term 0.
I20260619 06:41:07.085183 18347 raft_consensus.cc:2454] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ace7e56e4b4f43bb1e0d51c4ecebb1 in term 0.
I20260619 06:41:07.085556 18415 leader_election.cc:302] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ace7e56e4b4f43bb1e0d51c4ecebb1, 4408951f5cc643ae972810e7e3a7e793; no voters:
I20260619 06:41:07.085922 18551 raft_consensus.cc:2781] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:41:07.086040 18551 raft_consensus.cc:482] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:41:07.086105 18551 raft_consensus.cc:3037] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:41:07.088191 18551 raft_consensus.cc:504] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408951f5cc643ae972810e7e3a7e793" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44945 } } peers { permanent_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45829 } } peers { permanent_uuid: "4d1ee36c72ed4be6a61edf843195e758" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40529 } }
I20260619 06:41:07.088783 18551 leader_election.cc:288] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [CANDIDATE]: Term 1 election: Requested vote from peers 4408951f5cc643ae972810e7e3a7e793 (127.7.146.193:44945), 4d1ee36c72ed4be6a61edf843195e758 (127.7.146.194:40529)
I20260619 06:41:07.089183 18347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18ba4d85a6f44a6b9852d54b734bd3c" candidate_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d1ee36c72ed4be6a61edf843195e758"
I20260619 06:41:07.089350 18347 raft_consensus.cc:3037] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:41:07.089530 18213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18ba4d85a6f44a6b9852d54b734bd3c" candidate_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4408951f5cc643ae972810e7e3a7e793"
I20260619 06:41:07.089735 18213 raft_consensus.cc:3037] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:41:07.092411 18347 raft_consensus.cc:2454] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ace7e56e4b4f43bb1e0d51c4ecebb1 in term 1.
I20260619 06:41:07.092653 18213 raft_consensus.cc:2454] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ace7e56e4b4f43bb1e0d51c4ecebb1 in term 1.
I20260619 06:41:07.092824 18413 leader_election.cc:302] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ace7e56e4b4f43bb1e0d51c4ecebb1, 4d1ee36c72ed4be6a61edf843195e758; no voters:
I20260619 06:41:07.093127 18551 raft_consensus.cc:2781] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:41:07.093575 18551 raft_consensus.cc:686] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 1 LEADER]: Becoming Leader. State: Replica: 27ace7e56e4b4f43bb1e0d51c4ecebb1, State: Running, Role: LEADER
I20260619 06:41:07.094159 18551 consensus_queue.cc:237] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408951f5cc643ae972810e7e3a7e793" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44945 } } peers { permanent_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45829 } } peers { permanent_uuid: "4d1ee36c72ed4be6a61edf843195e758" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40529 } }
I20260619 06:41:07.098739 18069 catalog_manager.cc:5495] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27ace7e56e4b4f43bb1e0d51c4ecebb1 (127.7.146.195). New cstate: current_term: 1 leader_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4408951f5cc643ae972810e7e3a7e793" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ace7e56e4b4f43bb1e0d51c4ecebb1" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d1ee36c72ed4be6a61edf843195e758" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40529 } health_report { overall_health: UNKNOWN } } }
I20260619 06:41:07.129627 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:07.124632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.7.146.195:45829)
0619 06:41:07.124836 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:07.124841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:07.124907 (+ 66us) server_negotiation.cc:407] Connection header received
0619 06:41:07.125078 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:07.125082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:07.125156 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:07.125283 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:41:07.125657 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.126353 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.127558 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.127951 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.128071 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.128535 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:41:07.129079 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:41:07.129223 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:07.129294 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 06:41:07.129386 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:41:07.137661 18347 raft_consensus.cc:1261] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758 [term 1 FOLLOWER]: Refusing update from remote peer 27ace7e56e4b4f43bb1e0d51c4ecebb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:41:07.137662 18213 raft_consensus.cc:1261] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793 [term 1 FOLLOWER]: Refusing update from remote peer 27ace7e56e4b4f43bb1e0d51c4ecebb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:41:07.138608 18553 consensus_queue.cc:1035] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4408951f5cc643ae972810e7e3a7e793" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:41:07.138939 18551 consensus_queue.cc:1035] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d1ee36c72ed4be6a61edf843195e758" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:41:07.165545 18069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:41:07.165814 18069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:41:07.169291 18069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab4dd62cb8ea42b6907702dc400e8196: Sending DeleteTablet for 3 replicas of tablet c18ba4d85a6f44a6b9852d54b734bd3c
I20260619 06:41:07.171142 18460 tablet_service.cc:1514] Processing DeleteTablet for tablet c18ba4d85a6f44a6b9852d54b734bd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:41:07 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:35064
I20260619 06:41:07.171185 18193 tablet_service.cc:1514] Processing DeleteTablet for tablet c18ba4d85a6f44a6b9852d54b734bd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:41:07 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:33628
I20260619 06:41:07.172019 18327 tablet_service.cc:1514] Processing DeleteTablet for tablet c18ba4d85a6f44a6b9852d54b734bd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:41:07 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:51012
I20260619 06:41:07.173496 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 18130
I20260619 06:41:07.173848 18567 tablet_replica.cc:330] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793: stopping tablet replica
I20260619 06:41:07.174290 18567 raft_consensus.cc:2229] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:41:07.174870 18567 raft_consensus.cc:2258] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:41:07.175662 18566 tablet_replica.cc:330] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1: stopping tablet replica
I20260619 06:41:07.176221 18568 tablet_replica.cc:330] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758: stopping tablet replica
I20260619 06:41:07.176542 18568 raft_consensus.cc:2229] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:41:07.176951 18568 raft_consensus.cc:2258] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:41:07.177452 18566 raft_consensus.cc:2229] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:41:07.178053 18566 raft_consensus.cc:2258] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:41:07.178118 18567 ts_tablet_manager.cc:1905] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:41:07.178997 18568 ts_tablet_manager.cc:1905] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:41:07.179633 18566 ts_tablet_manager.cc:1905] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:41:07.185446 18568 ts_tablet_manager.cc:1918] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:41:07.185447 18567 ts_tablet_manager.cc:1918] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:41:07.185599 18567 log.cc:1196] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/c18ba4d85a6f44a6b9852d54b734bd3c
I20260619 06:41:07.185672 18568 log.cc:1196] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/c18ba4d85a6f44a6b9852d54b734bd3c
I20260619 06:41:07.186167 18567 ts_tablet_manager.cc:1939] T c18ba4d85a6f44a6b9852d54b734bd3c P 4408951f5cc643ae972810e7e3a7e793: Deleting consensus metadata
I20260619 06:41:07.186230 18566 ts_tablet_manager.cc:1918] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:41:07.186321 18568 ts_tablet_manager.cc:1939] T c18ba4d85a6f44a6b9852d54b734bd3c P 4d1ee36c72ed4be6a61edf843195e758: Deleting consensus metadata
I20260619 06:41:07.186425 18566 log.cc:1196] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/c18ba4d85a6f44a6b9852d54b734bd3c
I20260619 06:41:07.186939 18566 ts_tablet_manager.cc:1939] T c18ba4d85a6f44a6b9852d54b734bd3c P 27ace7e56e4b4f43bb1e0d51c4ecebb1: Deleting consensus metadata
I20260619 06:41:07.188129 18055 catalog_manager.cc:4841] TS 4408951f5cc643ae972810e7e3a7e793 (127.7.146.193:44945): tablet c18ba4d85a6f44a6b9852d54b734bd3c (table test-table [id=92d4f9092dee49e69a5a1c4487915992]) successfully deleted
I20260619 06:41:07.188473 18056 catalog_manager.cc:4841] TS 27ace7e56e4b4f43bb1e0d51c4ecebb1 (127.7.146.195:45829): tablet c18ba4d85a6f44a6b9852d54b734bd3c (table test-table [id=92d4f9092dee49e69a5a1c4487915992]) successfully deleted
I20260619 06:41:07.188695 18053 catalog_manager.cc:4841] TS 4d1ee36c72ed4be6a61edf843195e758 (127.7.146.194:40529): tablet c18ba4d85a6f44a6b9852d54b734bd3c (table test-table [id=92d4f9092dee49e69a5a1c4487915992]) successfully deleted
Jun 19 06:41:07 dist-test-slave-qfk4 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:41:07.206039 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:07.175058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.7.146.193:44945)
0619 06:41:07.175223 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:07.175230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:07.191122 (+ 15892us) server_negotiation.cc:407] Connection header received
0619 06:41:07.191273 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:07.191276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:07.191350 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:07.191432 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:07.193013 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.193782 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.195616 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.196094 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.196187 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.200799 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:07.200812 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:07.200828 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:07.200862 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:07.203448 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:07.204081 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:07.204088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:07.204090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:07.204208 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:07.204609 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:07.204614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:07.204617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:07.204960 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:07.205244 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:07.205513 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 06:41:07.205722 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:41:07.206666 18253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:07.321848 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 18130
I20260619 06:41:07.337091 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 18264
Jun 19 06:41:07 dist-test-slave-qfk4 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:41:07.354161 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:07.337837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.7.146.194:40529)
0619 06:41:07.337997 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:07.338002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:07.338020 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:41:07.338224 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:07.338227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:07.338292 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:07.338364 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:07.339900 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.340665 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.342409 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.342796 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.342899 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.347641 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:07.347661 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:07.347681 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:07.347731 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:07.351245 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:07.351933 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:07.351944 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:07.351947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:07.352101 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:07.352415 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:07.352419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:07.352423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:07.352945 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:07.353231 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:07.353569 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 06:41:07.353789 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:41:07.355145 18387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:07.452008 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 18264
I20260619 06:41:07.465157 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 18397
Jun 19 06:41:07 dist-test-slave-qfk4 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:41:07.481654 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:07.465917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.7.146.195:45829)
0619 06:41:07.466122 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:07.466128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:07.466152 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:41:07.466224 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:07.466230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:07.466312 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:07.466438 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:07.467672 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.468950 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.470258 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:07.470717 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:07.470834 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:07.475501 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:07.475517 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:07.475536 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:07.475581 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:07.478628 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:07.479525 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:07.479532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:07.479535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:07.479735 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:07.480167 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:07.480175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:07.480177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:07.480550 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:07.480831 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:07.481134 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 06:41:07.481320 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:41:07.482383 18520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:07.601015 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 18397
I20260619 06:41:07.616267 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 18037
I20260619 06:41:07.617395 18099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:07.724900 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 18037
2026-06-19T06:41:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4016 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:41:07 dist-test-slave-qfk4 krb5kdc[18576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:41:07 dist-test-slave-qfk4 krb5kdc[18576](info): set up 2 sockets
Jun 19 06:41:07 dist-test-slave-qfk4 krb5kdc[18576](info): commencing operation
krb5kdc: starting...
W20260619 06:41:08.745421 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:41:08 dist-test-slave-qfk4 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:41:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:41:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.146.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.146.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/rangeradmin_127.7.146.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/rangeradmin_127.7.146.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.146.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.146.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/rangerlookup_127.7.146.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/rangerlookup_127.7.146.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.146.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.146.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/HTTP_127.7.146.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/HTTP_127.7.146.212@KRBTEST.COM.keytab.
I20260619 06:41:09.045593 7755 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjzr1uN/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/postgres -l logfile start
2026-06-19 06:41:11.540 UTC [18606] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:41:11.540 UTC [18606] LOG: listening on IPv4 address "127.7.146.212", port 33701
2026-06-19 06:41:11.543 UTC [18606] LOG: listening on Unix socket "/tmp/.s.PGSQL.33701"
2026-06-19 06:41:11.563 UTC [18609] LOG: database system was shut down at 2026-06-19 06:41:10 UTC
2026-06-19 06:41:11.567 UTC [18606] LOG: database system is ready to accept connections
I20260619 06:41:12.468919 7755 mini_postgres.cc:96] Postgres bound to 33701
2026-06-19 06:41:12.485 UTC [18617] FATAL: database "slave" does not exist
127.7.146.212:33701 - accepting connections
I20260619 06:41:12.485836 7755 mini_ranger.cc:161] Starting Ranger...
I20260619 06:41:12.511036 7755 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:41:12.997051 7755 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:41:12.997175 7755 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-admin
2026-06-19 06:41:13,267 [I] DB FLAVOR :POSTGRES
2026-06-19 06:41:13,267 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:41:13,267 [I] Checking connection..
2026-06-19 06:41:13,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:41:13,521 [I] Checking connection passed.
2026-06-19 06:41:13,521 [I] --------- Verifying version history table ---------
2026-06-19 06:41:13,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:41:13,780 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:41:13,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:41:14,021 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:41:14,021 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:41:14,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:41:14.266 UTC [18681] WARNING: there is no transaction in progress
2026-06-19 06:41:14,271 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:41:14,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:41:14,512 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:41:14,512 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:41:14,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:41:14,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:41:14,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qfk4.c.gcp-upstream.internal','N') ;"
2026-06-19 06:41:15,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:41:15,502 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:41:15,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:41:15,764 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:41:15,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:41:16,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:41:16,264 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:41:16,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:41:17.237 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.254 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.272 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.288 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.305 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.323 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.382 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.390 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.405 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.416 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.427 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.441 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.451 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.458 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.465 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:17.472 UTC [18789] WARNING: there is no transaction in progress
2026-06-19 06:41:18,157 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:41:18,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:41:18,411 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:41:18,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:41:18,649 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:41:18,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:41:18,908 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:41:18,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:41:19,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:41:19,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qfk4.c.gcp-upstream.internal';"
2026-06-19 06:41:19,625 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:41:19,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:41:19,877 [I] DB_PATCHES have already been applied
I20260619 06:41:19.883366 7755 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjzr1uN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:41:19.883459 7755 mini_ranger.cc:193] Using host: 127.7.146.212
I20260619 06:41:19.897192 7755 mini_ranger.cc:227] Ranger admin URL: http://127.7.146.212:51697
Jun 19, 2026 6:41:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:41:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:41:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:41:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:41:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:41:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:41:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:41:20.931598 7755 mini_ranger.cc:160] Time spent starting Ranger: real 8.446s user 0.002s sys 0.036s
Jun 19, 2026 6:41:40 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:41:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:41:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:41:40 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:41:41 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 06:41:41.697654 7755 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.146.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.146.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/rangerkms_127.7.146.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/rangerkms_127.7.146.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.146.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.146.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.146.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/HTTP_127.7.146.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.146.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/HTTP_127.7.146.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 06:41:41 dist-test-slave-qfk4 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851301, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 06:41:41.884109 7755 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 06:41:41.903168 7755 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 06:41:42.746764 7755 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 06:41:42.746905 7755 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-kms
I20260619 06:41:42.747020 7755 mini_ranger_kms.cc:227] Using postgres at 127.7.146.212:33701
2026-06-19 06:41:43,132 [I] DB FLAVOR :POSTGRES
2026-06-19 06:41:43,132 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:41:43,132 [I] Checking connection
2026-06-19 06:41:43,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 06:41:43,360 [I] connection success
2026-06-19 06:41:43,360 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 06:41:43,360 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 06:41:43,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 06:41:43,602 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 06:41:43,602 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:41:43,602 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 06:41:43,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:33701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 06:41:43,854 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 06:41:44.325351 7755 mini_ranger_kms.cc:310] Created kms service
I20260619 06:41:44.526172 7755 mini_ranger_kms.cc:326] Created kudu user
I20260619 06:41:44.570118 7755 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 06:41:44 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851301, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:41:44.965173 7755 mini_ranger_kms.cc:384] Added ranger policy
I20260619 06:41:44.965518 7755 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 06:41:44.965590 7755 mini_ranger_kms.cc:240] Using host: 127.7.146.212
I20260619 06:41:44.981316 7755 mini_ranger_kms.cc:276] Ranger KMS PID: 18969
I20260619 06:41:44.981506 7755 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.146.212:58119
Jun 19, 2026 6:41:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:41:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:41:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:41:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:41:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:41:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 06:41:47.084436 7755 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.200s user 0.108s sys 0.295s
I20260619 06:41:47.084692 7755 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 06:41:47.084894 7755 mini_ranger_kms.cc:401] 127.7.146.212:58119/kms/v1/keys
Jun 19 06:41:47 dist-test-slave-qfk4 krb5kdc[18576](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851307, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.146.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 06:41:47 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851307, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.146.212@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
Jun 19 06:41:48 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851307, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.146.212@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
Jun 19 06:41:50 dist-test-slave-qfk4 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:41:50.393992 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40473
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:40473
--ranger_config_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:58119/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:41:50.714118 19020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:50.714413 19020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:50.714510 19020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:50.721987 19020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:41:50.722101 19020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:50.722143 19020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:50.722195 19020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:41:50.722239 19020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:41:50.732314 19020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.146.212:58119/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:40473
--ranger_config_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:50.733634 19020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:50.734899 19020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:50.746248 19025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:50.746284 19026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:50.746337 19028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:50.746619 19020 server_base.cc:1034] running on GCE node
I20260619 06:41:50.747618 19020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:50.749032 19020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:50.750243 19020 hybrid_clock.cc:648] HybridClock initialized: now 1781851310750200 us; error 30 us; skew 500 ppm
I20260619 06:41:50.750602 19020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:41:50 dist-test-slave-qfk4 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851310, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:41:50.753588 19020 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:41:50.754985 19020 webserver.cc:460] Webserver started at http://127.7.146.254:33837/ using document root <none> and password file <none>
I20260619 06:41:50.755666 19020 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:50.755764 19020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:50.756093 19020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:41:50 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851310, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:41:50.822337 19020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "8a8785613e2f46cd8baa1a85558ab0bf"
format_stamp: "Formatted at 2026-06-19 06:41:50 on dist-test-slave-qfk4"
server_key: "34b1864a1790f2439c31cfab78b4427d"
server_key_iv: "b8372ba5e0d41eec815c70e01bc5a615"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:50.823443 19020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "8a8785613e2f46cd8baa1a85558ab0bf"
format_stamp: "Formatted at 2026-06-19 06:41:50 on dist-test-slave-qfk4"
server_key: "34b1864a1790f2439c31cfab78b4427d"
server_key_iv: "b8372ba5e0d41eec815c70e01bc5a615"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:50.830519 19020 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260619 06:41:50.851531 19035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:50.853408 19020 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:41:50.853597 19020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "8a8785613e2f46cd8baa1a85558ab0bf"
format_stamp: "Formatted at 2026-06-19 06:41:50 on dist-test-slave-qfk4"
server_key: "34b1864a1790f2439c31cfab78b4427d"
server_key_iv: "b8372ba5e0d41eec815c70e01bc5a615"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:50.853848 19020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:50.875625 19020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:50.880657 19020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:50.880966 19020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:50.900617 19020 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:40473
I20260619 06:41:50.900621 19086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:40473 every 8 connection(s)
I20260619 06:41:50.902328 19020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:41:50.906205 19087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:41:50.908640 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 19020
I20260619 06:41:50.908838 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:41:50.909582 7755 external_mini_cluster.cc:1428] Setting key 1e9bac603dbad869b61be581529e6857
I20260619 06:41:50.917846 19087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf: Bootstrap starting.
I20260619 06:41:50.921048 19087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf: Neither blocks nor log segments found. Creating new log.
I20260619 06:41:50.922008 19087 log.cc:824] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:41:50 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:41:50.925104 19087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf: No bootstrap required, opened a new log
I20260619 06:41:50.930420 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:50.911783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45232 (local address 127.7.146.254:40473)
0619 06:41:50.912548 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:50.912596 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:50.912651 (+ 55us) server_negotiation.cc:407] Connection header received
0619 06:41:50.914252 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:50.914274 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:50.914670 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:50.915030 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:50.916308 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:50.917618 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:50.918860 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:50.919575 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:50.919741 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:50.923644 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:50.923652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:50.923668 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:50.923702 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:50.926550 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:50.927239 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:50.927245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:50.927251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:50.927410 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:50.927771 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:50.927774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:50.927776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:50.928286 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:50.928547 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:50.929031 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 06:41:50.929407 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":324,"threads_started":1}
I20260619 06:41:50.933095 19087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8785613e2f46cd8baa1a85558ab0bf" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40473 } }
I20260619 06:41:50.933529 19087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:41:50.933616 19087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8785613e2f46cd8baa1a85558ab0bf, State: Initialized, Role: FOLLOWER
I20260619 06:41:50.934147 19087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8785613e2f46cd8baa1a85558ab0bf" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40473 } }
I20260619 06:41:50.934355 19087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:41:50.934487 19087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:41:50.934623 19087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:41:50.936610 19087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8785613e2f46cd8baa1a85558ab0bf" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40473 } }
I20260619 06:41:50.937134 19087 leader_election.cc:302] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a8785613e2f46cd8baa1a85558ab0bf; no voters:
I20260619 06:41:50.937740 19087 leader_election.cc:288] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:41:50.937837 19092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:41:50.938416 19092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [term 1 LEADER]: Becoming Leader. State: Replica: 8a8785613e2f46cd8baa1a85558ab0bf, State: Running, Role: LEADER
I20260619 06:41:50.938920 19092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8785613e2f46cd8baa1a85558ab0bf" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40473 } }
I20260619 06:41:50.939615 19087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:41:50.941742 19094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a8785613e2f46cd8baa1a85558ab0bf. Latest consensus state: current_term: 1 leader_uuid: "8a8785613e2f46cd8baa1a85558ab0bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8785613e2f46cd8baa1a85558ab0bf" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40473 } } }
I20260619 06:41:50.942148 19094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [sys.catalog]: This master's current role is: LEADER
I20260619 06:41:50.942976 19101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:41:50.941702 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a8785613e2f46cd8baa1a85558ab0bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8785613e2f46cd8baa1a85558ab0bf" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40473 } } }
I20260619 06:41:50.944041 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf [sys.catalog]: This master's current role is: LEADER
I20260619 06:41:50.948747 19101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:41:50.949309 19087 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:41:50.967686 19101 catalog_manager.cc:1347] Generated new cluster ID: 6905bce6beec453698079d53d73c4d83
I20260619 06:41:50.967832 19101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:41:50.987032 19101 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:41:50.989435 19101 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:41:50.999135 19101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf: Generated new TSK 0
I20260619 06:41:51.000274 19101 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:41:51.734081 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:58119/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40473
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:41:52 dist-test-slave-qfk4 krb5kdc[18576](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851312, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 06:41:52.120411 19132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:52.120849 19132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:52.121007 19132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:52.132622 19132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:52.132766 19132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:52.133009 19132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:41:52.148806 19132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.146.212:58119/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:52.150720 19132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:52.152400 19132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:52.185331 19143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:52.186276 19142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:52.189446 19145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:52.190253 19132 server_base.cc:1034] running on GCE node
I20260619 06:41:52.191072 19132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:52.192381 19132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:52.193653 19132 hybrid_clock.cc:648] HybridClock initialized: now 1781851312193582 us; error 55 us; skew 500 ppm
I20260619 06:41:52.194133 19132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:41:52 dist-test-slave-qfk4 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851312, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:41:52.198422 19132 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:41:52.202064 19132 webserver.cc:460] Webserver started at http://127.7.146.193:43463/ using document root <none> and password file <none>
I20260619 06:41:52.202975 19132 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:52.203117 19132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:52.203469 19132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:41:52 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851312, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:41:52.225330 19132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "3207c453a19b44f99ad28a13a8757fa3"
format_stamp: "Formatted at 2026-06-19 06:41:52 on dist-test-slave-qfk4"
server_key: "b2c79ad4ec18f98296ee972fd22e8ba1"
server_key_iv: "5a9883db20e45806796e7d38598f2945"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:52.226447 19132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "3207c453a19b44f99ad28a13a8757fa3"
format_stamp: "Formatted at 2026-06-19 06:41:52 on dist-test-slave-qfk4"
server_key: "b2c79ad4ec18f98296ee972fd22e8ba1"
server_key_iv: "5a9883db20e45806796e7d38598f2945"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:52.251053 19132 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.014s sys 0.000s
I20260619 06:41:52.276362 19152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:52.278620 19132 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260619 06:41:52.278837 19132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "3207c453a19b44f99ad28a13a8757fa3"
format_stamp: "Formatted at 2026-06-19 06:41:52 on dist-test-slave-qfk4"
server_key: "b2c79ad4ec18f98296ee972fd22e8ba1"
server_key_iv: "5a9883db20e45806796e7d38598f2945"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:52.279140 19132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:52.301570 19132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:52.307299 19132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:52.307719 19132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:52.315328 19132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:52.327178 19132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:52.327389 19132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 06:41:52.327600 19132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:52.327725 19132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:52.400367 19132 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:37953
I20260619 06:41:52.400403 19264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:37953 every 8 connection(s)
I20260619 06:41:52.402287 19132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:41:52.403878 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 19132
I20260619 06:41:52.404067 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:41:52.404919 7755 external_mini_cluster.cc:1428] Setting key 98edb0fec632d3a8bcc4bd05f804a18b
Jun 19 06:41:52 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851312, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
I20260619 06:41:52.451997 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:52.412183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40473 (local address 127.7.146.193:59765)
0619 06:41:52.427971 (+ 15788us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:52.428036 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:52.430679 (+ 2643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:52.431899 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:52.431913 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:52.432616 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:52.433545 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:52.433574 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:52.435186 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:52.435191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:52.437318 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:52.437333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:52.438660 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:52.438668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:52.438990 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:52.439880 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:41:52.439914 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:52.442727 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:52.446813 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:52.446823 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:52.446843 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:52.447515 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:52.448008 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:52.448012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:52.448016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:52.448322 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:52.449052 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:52.449060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:52.449538 (+ 478us) client_negotiation.cc:769] Sending connection context
0619 06:41:52.449936 (+ 398us) client_negotiation.cc:240] Negotiation successful
0619 06:41:52.450517 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15430,"thread_start_us":15237,"threads_started":1}
I20260619 06:41:52.453917 19268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:52.412796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:59765 (local address 127.7.146.254:40473)
0619 06:41:52.427305 (+ 14509us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:52.427312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:52.428682 (+ 1370us) server_negotiation.cc:407] Connection header received
0619 06:41:52.431069 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:52.431076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:52.431188 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:52.431275 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:52.433767 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:52.434952 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:52.437496 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:52.438188 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:52.438293 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:52.443038 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:52.443045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:52.443050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:52.443093 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:52.446618 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:52.447675 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:52.447681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:52.447684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:52.447851 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:52.448461 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:52.448466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:52.448470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:52.448850 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:52.448979 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:52.450144 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0619 06:41:52.450499 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14365,"thread_start_us":14224,"threads_started":1}
I20260619 06:41:52.455871 19265 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40473
I20260619 06:41:52.456218 19265 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:52.457207 19265 heartbeater.cc:511] Master 127.7.146.254:40473 requested a full tablet report, sending...
I20260619 06:41:52.460044 19052 ts_manager.cc:194] Registered new tserver with Master: 3207c453a19b44f99ad28a13a8757fa3 (127.7.146.193:37953)
I20260619 06:41:52.462126 19052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:59765
Jun 19 06:41:52 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851312, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:41:52.508193 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:58119/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40473
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:41:52 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851312, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
W20260619 06:41:52.982900 19271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:52.983414 19271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:52.983611 19271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:52.995025 19271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:52.995270 19271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:52.995507 19271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:41:53.011240 19271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.146.212:58119/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:53.013096 19271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:53.014869 19271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:53.046226 19279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:53.051467 19278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:53.051918 19271 server_base.cc:1034] running on GCE node
W20260619 06:41:53.052925 19281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:53.053777 19271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:53.055114 19271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:53.056396 19271 hybrid_clock.cc:648] HybridClock initialized: now 1781851313056331 us; error 52 us; skew 500 ppm
I20260619 06:41:53.056825 19271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:41:53 dist-test-slave-qfk4 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851313, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:41:53.061120 19271 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:41:53.066088 19271 webserver.cc:460] Webserver started at http://127.7.146.194:42705/ using document root <none> and password file <none>
I20260619 06:41:53.066984 19271 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:53.067114 19271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:53.067451 19271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:41:53 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851313, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:41:53.087430 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a"
format_stamp: "Formatted at 2026-06-19 06:41:53 on dist-test-slave-qfk4"
server_key: "c9dbf85ad42579f3095fb1bc62cdce4b"
server_key_iv: "f97aa4c352d38ffa56584db030672003"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:53.089026 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a"
format_stamp: "Formatted at 2026-06-19 06:41:53 on dist-test-slave-qfk4"
server_key: "c9dbf85ad42579f3095fb1bc62cdce4b"
server_key_iv: "f97aa4c352d38ffa56584db030672003"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:53.096215 19271 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 06:41:53.118531 19293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:53.120273 19271 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260619 06:41:53.120455 19271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a"
format_stamp: "Formatted at 2026-06-19 06:41:53 on dist-test-slave-qfk4"
server_key: "c9dbf85ad42579f3095fb1bc62cdce4b"
server_key_iv: "f97aa4c352d38ffa56584db030672003"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:53.120724 19271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:53.157814 19271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:53.161893 19271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:53.162169 19271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:53.163154 19271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:53.164748 19271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:53.164825 19271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:53.164947 19271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:53.164999 19271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:53.204872 19271 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:40955
I20260619 06:41:53.204897 19407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:40955 every 8 connection(s)
I20260619 06:41:53.206249 19271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:41:53.211410 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 19271
I20260619 06:41:53.211611 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:41:53.212255 7755 external_mini_cluster.cc:1428] Setting key e3f1d270fe0f53d923759b9648e7e461
Jun 19 06:41:53 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851313, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:41:53.239409 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:53.209256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:41957 (local address 127.7.146.254:40473)
0619 06:41:53.217458 (+ 8202us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:53.217464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:53.218545 (+ 1081us) server_negotiation.cc:407] Connection header received
0619 06:41:53.220739 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:53.220745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:53.220835 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:53.220927 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:53.222965 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.223989 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.226288 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.226883 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.226976 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:53.231641 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:53.231648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:53.231653 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:53.231693 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:53.235213 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:53.236226 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:53.236232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:53.236236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:53.236398 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:53.237044 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:53.237050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:53.237054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:53.237486 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:53.237643 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:53.238685 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0619 06:41:53.238937 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8058,"thread_start_us":8046,"threads_started":1}
I20260619 06:41:53.241984 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:53.210001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40473 (local address 127.7.146.194:41957)
0619 06:41:53.217996 (+ 7995us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:53.218045 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:53.220305 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:53.221357 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:53.221369 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:53.221967 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:53.222773 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:53.222797 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.224187 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.224192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:53.226128 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:53.226142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.227316 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.227324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:53.227642 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:53.228475 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:41:53.228504 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:53.231322 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:53.235371 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:53.235382 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:53.235401 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:53.236061 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:53.236592 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:53.236598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:53.236602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:53.236878 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:53.237733 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:53.237742 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:53.238190 (+ 448us) client_negotiation.cc:769] Sending connection context
0619 06:41:53.239741 (+ 1551us) client_negotiation.cc:240] Negotiation successful
0619 06:41:53.240348 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7700,"thread_start_us":7467,"threads_started":1}
I20260619 06:41:53.243806 19408 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40473
I20260619 06:41:53.244176 19408 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:53.245126 19408 heartbeater.cc:511] Master 127.7.146.254:40473 requested a full tablet report, sending...
I20260619 06:41:53.246893 19052 ts_manager.cc:194] Registered new tserver with Master: 9a02c16b5dff4dc8a5101c62a9bb4e9a (127.7.146.194:40955)
I20260619 06:41:53.248099 19052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:41957
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:41:53.288347 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:58119/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40473
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:41:53.466503 19265 heartbeater.cc:503] Master 127.7.146.254:40473 was elected leader, sending a full tablet report...
W20260619 06:41:53.606823 19414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:41:53.607120 19414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:41:53.607244 19414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:41:53.614482 19414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:41:53.614609 19414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:41:53.614758 19414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:41:53.624476 19414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.146.212:58119/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:41:53.625800 19414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:41:53.627017 19414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:41:53.639518 19419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:53.639551 19420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:41:53.639582 19422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:41:53.640136 19414 server_base.cc:1034] running on GCE node
I20260619 06:41:53.640844 19414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:41:53.641794 19414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:41:53.643030 19414 hybrid_clock.cc:648] HybridClock initialized: now 1781851313642965 us; error 56 us; skew 500 ppm
I20260619 06:41:53.643370 19414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:41:53 dist-test-slave-qfk4 krb5kdc[18576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851313, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:41:53.646812 19414 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:41:53.648443 19414 webserver.cc:460] Webserver started at http://127.7.146.195:34381/ using document root <none> and password file <none>
I20260619 06:41:53.649199 19414 fs_manager.cc:359] Metadata directory not provided
I20260619 06:41:53.649325 19414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:41:53.649648 19414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:41:53 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851313, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:41:53.667630 19414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "2aebef1fc72e4210b975f7c3851f4247"
format_stamp: "Formatted at 2026-06-19 06:41:53 on dist-test-slave-qfk4"
server_key: "6a363cf06f542a64a1b0888ad96376e7"
server_key_iv: "f4a5eca58c888534ce708d59579bfa03"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:53.668797 19414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "2aebef1fc72e4210b975f7c3851f4247"
format_stamp: "Formatted at 2026-06-19 06:41:53 on dist-test-slave-qfk4"
server_key: "6a363cf06f542a64a1b0888ad96376e7"
server_key_iv: "f4a5eca58c888534ce708d59579bfa03"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:53.675230 19414 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 06:41:53.690160 19429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:53.691937 19414 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:41:53.692121 19414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "2aebef1fc72e4210b975f7c3851f4247"
format_stamp: "Formatted at 2026-06-19 06:41:53 on dist-test-slave-qfk4"
server_key: "6a363cf06f542a64a1b0888ad96376e7"
server_key_iv: "f4a5eca58c888534ce708d59579bfa03"
server_key_version: "kuduclusterkey@0"
I20260619 06:41:53.692373 19414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:41:53.706506 19414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:41:53.710094 19414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:41:53.710332 19414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:41:53.711190 19414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:41:53.712474 19414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:41:53.712543 19414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:53.712675 19414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:41:53.712726 19414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:41:53.742062 19414 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:39941
I20260619 06:41:53.742126 19541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:39941 every 8 connection(s)
I20260619 06:41:53.743253 19414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:41:53.752344 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 19414
I20260619 06:41:53.752556 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:41:53.753114 7755 external_mini_cluster.cc:1428] Setting key 401c16da457e004e8b9aa2a0f3495ccd
Jun 19 06:41:53 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851313, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:41:53.762684 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:53.745806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:33939 (local address 127.7.146.254:40473)
0619 06:41:53.746278 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:53.746282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:53.747203 (+ 921us) server_negotiation.cc:407] Connection header received
0619 06:41:53.749011 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:53.749015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:53.749081 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:53.749175 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:53.750513 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.751246 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.752814 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.753430 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.753558 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:53.757220 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:53.757225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:53.757229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:53.757263 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:53.759992 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:53.760726 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:53.760730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:53.760732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:53.760853 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:53.761189 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:53.761192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:53.761194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:53.761449 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:53.761552 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:53.762215 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 06:41:53.762398 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":223,"threads_started":1}
I20260619 06:41:53.763553 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:53.746375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40473 (local address 127.7.146.195:33939)
0619 06:41:53.747038 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:53.747080 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:53.748742 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:53.749359 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:53.749367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:53.749779 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:41:53.750356 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:53.750372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.751357 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.751360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:53.752679 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:53.752691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.753639 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.753644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:53.753930 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:53.754634 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:41:53.754658 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:41:53.757016 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:41:53.760096 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:53.760103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:53.760116 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:53.760630 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:53.760935 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:41:53.760938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:41:53.760940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:41:53.761099 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:41:53.761539 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:41:53.761545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:41:53.761864 (+ 319us) client_negotiation.cc:769] Sending connection context
0619 06:41:53.762078 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 06:41:53.762476 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":273,"threads_started":1}
I20260619 06:41:53.764997 19542 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40473
I20260619 06:41:53.765285 19542 heartbeater.cc:464] Registering TS with master...
I20260619 06:41:53.765990 19542 heartbeater.cc:511] Master 127.7.146.254:40473 requested a full tablet report, sending...
I20260619 06:41:53.767378 19052 ts_manager.cc:194] Registered new tserver with Master: 2aebef1fc72e4210b975f7c3851f4247 (127.7.146.195:39941)
I20260619 06:41:53.768383 19052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:33939
I20260619 06:41:53.769062 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:41:53.791041 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:53.773505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.7.146.254:40473)
0619 06:41:53.773703 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:53.773709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:53.773975 (+ 266us) server_negotiation.cc:407] Connection header received
0619 06:41:53.774127 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:53.774131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:53.774202 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:53.774276 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:53.775567 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.776268 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.777554 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.777952 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.778056 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:53.786279 (+ 8223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:53.786285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:53.786289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:53.786326 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:53.788701 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:53.789354 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:53.789358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:53.789360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:53.789483 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:53.789817 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:53.789821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:53.789824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:53.790112 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:53.790291 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:53.790542 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 06:41:53.790719 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:41:53.794915 19052 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:41:53.801816 19052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:41:53.826268 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:53.813317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:39941 (local address 127.0.0.1:46826)
0619 06:41:53.814148 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:53.814191 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:53.814340 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:53.815861 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:53.815870 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:53.815905 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:41:53.816292 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:53.816304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.819479 (+ 3175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.819485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:53.823266 (+ 3781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:53.823285 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.825327 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.825334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:53.825577 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:53.825635 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 06:41:53.825735 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:41:53.825918 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":460,"threads_started":1}
I20260619 06:41:53.827445 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:53.813684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.7.146.195:39941)
0619 06:41:53.814953 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:53.814961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:53.814991 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:41:53.815079 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:53.815086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:53.815359 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:53.815541 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:41:53.817549 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.819294 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.823461 (+ 4167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.825087 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.826677 (+ 1590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:53.826900 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:53.826952 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 06:41:53.827099 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":956,"threads_started":1}
I20260619 06:41:53.831270 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:53.814946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.7.146.193:37953)
0619 06:41:53.823882 (+ 8936us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:53.823892 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:53.823921 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:41:53.824024 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:53.824031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:53.824326 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:53.825358 (+ 1032us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:41:53.825582 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.826678 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.828807 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.830223 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.830355 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:53.830583 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:53.830785 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 06:41:53.830940 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8778,"thread_start_us":8624,"threads_started":1}
I20260619 06:41:53.831269 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:53.814649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:37953 (local address 127.0.0.1:32780)
0619 06:41:53.815203 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:53.815224 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:53.815323 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:53.824547 (+ 9224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:53.824552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:53.824614 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:41:53.825441 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:53.825457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.826811 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.826817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:53.828670 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:53.828681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.830437 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.830443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:53.830604 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:53.830636 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:41:53.830732 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 06:41:53.830850 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":5,"thread_start_us":324,"threads_started":1}
I20260619 06:41:53.831768 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:53.815240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:40955 (local address 127.0.0.1:39978)
0619 06:41:53.821816 (+ 6576us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:53.821832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:53.821914 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:53.824775 (+ 2861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:53.824780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:53.824806 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:41:53.825279 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:53.825295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.827053 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.827058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:53.828814 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:53.828827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.830731 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.830737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:53.830893 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:53.830924 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:41:53.831022 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 06:41:53.831155 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6463,"mutex_wait_us":45,"thread_start_us":6306,"threads_started":1}
I20260619 06:41:53.831815 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:53.815702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.7.146.194:40955)
0619 06:41:53.824192 (+ 8490us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:53.824200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:53.824225 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:41:53.824312 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:53.824318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:53.824530 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:53.824777 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:41:53.825576 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.826925 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.828955 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:53.830332 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:53.830446 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:53.830586 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:53.831347 (+ 761us) server_negotiation.cc:299] Negotiation successful
0619 06:41:53.831507 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8335,"thread_start_us":8226,"threads_started":1}
I20260619 06:41:53.831480 19477 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec38b7d36a7421f834b1aad7e0f863e (DEFAULT_TABLE table=test-table [id=455ff64a44f946e0a563d0b12f607d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:41:53.833070 19477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec38b7d36a7421f834b1aad7e0f863e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:41:53.834969 19343 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec38b7d36a7421f834b1aad7e0f863e (DEFAULT_TABLE table=test-table [id=455ff64a44f946e0a563d0b12f607d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:41:53.834993 19200 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec38b7d36a7421f834b1aad7e0f863e (DEFAULT_TABLE table=test-table [id=455ff64a44f946e0a563d0b12f607d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:41:53.836257 19343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec38b7d36a7421f834b1aad7e0f863e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:41:53.836328 19200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec38b7d36a7421f834b1aad7e0f863e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:41:53.846933 19561 tablet_bootstrap.cc:492] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: Bootstrap starting.
I20260619 06:41:53.849890 19563 tablet_bootstrap.cc:492] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: Bootstrap starting.
I20260619 06:41:53.850756 19561 tablet_bootstrap.cc:654] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: Neither blocks nor log segments found. Creating new log.
I20260619 06:41:53.852741 19561 log.cc:824] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:41:53.853300 19563 tablet_bootstrap.cc:654] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: Neither blocks nor log segments found. Creating new log.
I20260619 06:41:53.853411 19562 tablet_bootstrap.cc:492] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: Bootstrap starting.
I20260619 06:41:53.854538 19563 log.cc:824] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:41:53.855954 19561 tablet_bootstrap.cc:492] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: No bootstrap required, opened a new log
I20260619 06:41:53.856235 19561 ts_tablet_manager.cc:1397] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260619 06:41:53.856760 19562 tablet_bootstrap.cc:654] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: Neither blocks nor log segments found. Creating new log.
I20260619 06:41:53.856817 19563 tablet_bootstrap.cc:492] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: No bootstrap required, opened a new log
I20260619 06:41:53.857038 19563 ts_tablet_manager.cc:1397] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:41:53.857849 19562 log.cc:824] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: Log is configured to *not* fsync() on all Append() calls
I20260619 06:41:53.860714 19562 tablet_bootstrap.cc:492] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: No bootstrap required, opened a new log
I20260619 06:41:53.860958 19562 ts_tablet_manager.cc:1397] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260619 06:41:53.864459 19563 raft_consensus.cc:348] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aebef1fc72e4210b975f7c3851f4247" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 39941 } } peers { permanent_uuid: "3207c453a19b44f99ad28a13a8757fa3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37953 } } peers { permanent_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40955 } }
I20260619 06:41:53.864843 19563 raft_consensus.cc:374] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:41:53.864975 19563 raft_consensus.cc:729] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3207c453a19b44f99ad28a13a8757fa3, State: Initialized, Role: FOLLOWER
I20260619 06:41:53.865523 19563 consensus_queue.cc:260] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aebef1fc72e4210b975f7c3851f4247" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 39941 } } peers { permanent_uuid: "3207c453a19b44f99ad28a13a8757fa3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37953 } } peers { permanent_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40955 } }
I20260619 06:41:53.866287 19561 raft_consensus.cc:348] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aebef1fc72e4210b975f7c3851f4247" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 39941 } } peers { permanent_uuid: "3207c453a19b44f99ad28a13a8757fa3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37953 } } peers { permanent_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40955 } }
I20260619 06:41:53.866670 19561 raft_consensus.cc:374] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:41:53.866760 19561 raft_consensus.cc:729] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a02c16b5dff4dc8a5101c62a9bb4e9a, State: Initialized, Role: FOLLOWER
I20260619 06:41:53.867405 19561 consensus_queue.cc:260] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aebef1fc72e4210b975f7c3851f4247" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 39941 } } peers { permanent_uuid: "3207c453a19b44f99ad28a13a8757fa3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37953 } } peers { permanent_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40955 } }
I20260619 06:41:53.868193 19563 ts_tablet_manager.cc:1428] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 06:41:53.869071 19408 heartbeater.cc:503] Master 127.7.146.254:40473 was elected leader, sending a full tablet report...
I20260619 06:41:53.869635 19561 ts_tablet_manager.cc:1428] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260619 06:41:53.870689 19562 raft_consensus.cc:348] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aebef1fc72e4210b975f7c3851f4247" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 39941 } } peers { permanent_uuid: "3207c453a19b44f99ad28a13a8757fa3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37953 } } peers { permanent_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40955 } }
I20260619 06:41:53.871080 19562 raft_consensus.cc:374] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:41:53.871214 19562 raft_consensus.cc:729] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aebef1fc72e4210b975f7c3851f4247, State: Initialized, Role: FOLLOWER
I20260619 06:41:53.871744 19562 consensus_queue.cc:260] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aebef1fc72e4210b975f7c3851f4247" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 39941 } } peers { permanent_uuid: "3207c453a19b44f99ad28a13a8757fa3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37953 } } peers { permanent_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40955 } }
I20260619 06:41:53.873221 19542 heartbeater.cc:503] Master 127.7.146.254:40473 was elected leader, sending a full tablet report...
I20260619 06:41:53.873683 19562 ts_tablet_manager.cc:1428] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
W20260619 06:41:53.918777 19266 tablet.cc:2369] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:41:53.967906 19409 tablet.cc:2369] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:41:53.994781 19543 tablet.cc:2369] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:41:54.060040 19568 raft_consensus.cc:482] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:41:54.060344 19568 raft_consensus.cc:504] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aebef1fc72e4210b975f7c3851f4247" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 39941 } } peers { permanent_uuid: "3207c453a19b44f99ad28a13a8757fa3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37953 } } peers { permanent_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40955 } }
I20260619 06:41:54.062143 19568 leader_election.cc:288] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aebef1fc72e4210b975f7c3851f4247 (127.7.146.195:39941), 3207c453a19b44f99ad28a13a8757fa3 (127.7.146.193:37953)
I20260619 06:41:54.069841 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:54.062542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:58951 (local address 127.7.146.195:39941)
0619 06:41:54.062755 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:54.062763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:54.063176 (+ 413us) server_negotiation.cc:407] Connection header received
0619 06:41:54.063352 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:54.063357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:54.063433 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:54.063517 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:41:54.064236 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.065242 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.067720 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.068921 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.069087 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:54.069161 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:54.069479 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 06:41:54.069600 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:41:54.069883 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:54.063392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:45169 (local address 127.7.146.193:37953)
0619 06:41:54.063613 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:54.063619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:54.063877 (+ 258us) server_negotiation.cc:407] Connection header received
0619 06:41:54.064024 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:54.064028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:54.064086 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:54.064161 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:41:54.064659 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.065563 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.067869 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.069055 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.069190 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:54.069241 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:54.069549 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 06:41:54.069677 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:41:54.069950 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:54.063134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:37953 (local address 127.7.146.194:45169)
0619 06:41:54.063784 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:54.063808 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:54.063920 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:54.064195 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:54.064199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:54.064222 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:41:54.064472 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:54.064479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.065715 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.065720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:54.067724 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:54.067733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.069235 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.069240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:54.069402 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:54.069430 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:41:54.069509 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:41:54.069605 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":362,"threads_started":1}
I20260619 06:41:54.070027 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:54.062476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:39941 (local address 127.7.146.194:58951)
0619 06:41:54.063058 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 06:41:54.063078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:41:54.063224 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:41:54.063688 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:41:54.063693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:41:54.063717 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:41:54.064097 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:54.064107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.065377 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.065381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:54.067550 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:41:54.067564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.069135 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.069140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:41:54.069312 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:54.069348 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:41:54.069441 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:41:54.069568 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":1,"thread_start_us":297,"threads_started":1}
I20260619 06:41:54.071058 19220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec38b7d36a7421f834b1aad7e0f863e" candidate_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3207c453a19b44f99ad28a13a8757fa3" is_pre_election: true
I20260619 06:41:54.071072 19497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec38b7d36a7421f834b1aad7e0f863e" candidate_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aebef1fc72e4210b975f7c3851f4247" is_pre_election: true
I20260619 06:41:54.071549 19220 raft_consensus.cc:2454] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a02c16b5dff4dc8a5101c62a9bb4e9a in term 0.
I20260619 06:41:54.071556 19497 raft_consensus.cc:2454] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a02c16b5dff4dc8a5101c62a9bb4e9a in term 0.
I20260619 06:41:54.072145 19299 leader_election.cc:302] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3207c453a19b44f99ad28a13a8757fa3, 9a02c16b5dff4dc8a5101c62a9bb4e9a; no voters:
I20260619 06:41:54.072511 19568 raft_consensus.cc:2781] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:41:54.072688 19568 raft_consensus.cc:482] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:41:54.072779 19568 raft_consensus.cc:3037] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:41:54.074832 19568 raft_consensus.cc:504] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aebef1fc72e4210b975f7c3851f4247" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 39941 } } peers { permanent_uuid: "3207c453a19b44f99ad28a13a8757fa3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37953 } } peers { permanent_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40955 } }
I20260619 06:41:54.075368 19568 leader_election.cc:288] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [CANDIDATE]: Term 1 election: Requested vote from peers 2aebef1fc72e4210b975f7c3851f4247 (127.7.146.195:39941), 3207c453a19b44f99ad28a13a8757fa3 (127.7.146.193:37953)
I20260619 06:41:54.075842 19497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec38b7d36a7421f834b1aad7e0f863e" candidate_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aebef1fc72e4210b975f7c3851f4247"
I20260619 06:41:54.075850 19220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec38b7d36a7421f834b1aad7e0f863e" candidate_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3207c453a19b44f99ad28a13a8757fa3"
I20260619 06:41:54.076032 19497 raft_consensus.cc:3037] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:41:54.076030 19220 raft_consensus.cc:3037] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:41:54.078121 19497 raft_consensus.cc:2454] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a02c16b5dff4dc8a5101c62a9bb4e9a in term 1.
I20260619 06:41:54.078122 19220 raft_consensus.cc:2454] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a02c16b5dff4dc8a5101c62a9bb4e9a in term 1.
I20260619 06:41:54.078500 19299 leader_election.cc:302] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aebef1fc72e4210b975f7c3851f4247, 9a02c16b5dff4dc8a5101c62a9bb4e9a; no voters:
I20260619 06:41:54.078795 19568 raft_consensus.cc:2781] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:41:54.079222 19568 raft_consensus.cc:686] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 1 LEADER]: Becoming Leader. State: Replica: 9a02c16b5dff4dc8a5101c62a9bb4e9a, State: Running, Role: LEADER
I20260619 06:41:54.079679 19568 consensus_queue.cc:237] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aebef1fc72e4210b975f7c3851f4247" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 39941 } } peers { permanent_uuid: "3207c453a19b44f99ad28a13a8757fa3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37953 } } peers { permanent_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40955 } }
I20260619 06:41:54.083941 19052 catalog_manager.cc:5495] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a02c16b5dff4dc8a5101c62a9bb4e9a (127.7.146.194). New cstate: current_term: 1 leader_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aebef1fc72e4210b975f7c3851f4247" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3207c453a19b44f99ad28a13a8757fa3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a02c16b5dff4dc8a5101c62a9bb4e9a" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40955 } health_report { overall_health: HEALTHY } } }
I20260619 06:41:54.117413 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:54.112257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.7.146.194:40955)
0619 06:41:54.112456 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:54.112462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:54.112481 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:41:54.112632 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:54.112636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:54.112711 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:54.112835 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:41:54.113246 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.113925 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.115363 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.115735 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.115837 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:54.116309 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:41:54.116835 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:41:54.116964 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:54.117068 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 06:41:54.117156 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:41:54.125533 19497 raft_consensus.cc:1261] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247 [term 1 FOLLOWER]: Refusing update from remote peer 9a02c16b5dff4dc8a5101c62a9bb4e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:41:54.125558 19220 raft_consensus.cc:1261] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3 [term 1 FOLLOWER]: Refusing update from remote peer 9a02c16b5dff4dc8a5101c62a9bb4e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:41:54.126435 19568 consensus_queue.cc:1035] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3207c453a19b44f99ad28a13a8757fa3" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:41:54.126812 19572 consensus_queue.cc:1035] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aebef1fc72e4210b975f7c3851f4247" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:41:54.138134 19575 mvcc.cc:205] Tried to move back new op lower bound from 7298462982647914496 to 7298462982473957376. Current Snapshot: MvccSnapshot[applied={T|T < 7298462982647914496}]
I20260619 06:41:54.151280 19051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:41:54.151563 19051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:41:54.155371 19051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a8785613e2f46cd8baa1a85558ab0bf: Sending DeleteTablet for 3 replicas of tablet 8ec38b7d36a7421f834b1aad7e0f863e
I20260619 06:41:54.156499 19477 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec38b7d36a7421f834b1aad7e0f863e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:41:54 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:46826
I20260619 06:41:54.156754 19200 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec38b7d36a7421f834b1aad7e0f863e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:41:54 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:32780
I20260619 06:41:54.156961 19343 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec38b7d36a7421f834b1aad7e0f863e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:41:54 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:39978
I20260619 06:41:54.159260 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 19132
I20260619 06:41:54.170622 19584 tablet_replica.cc:330] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: stopping tablet replica
I20260619 06:41:54.171231 19584 raft_consensus.cc:2229] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:41:54.171938 19584 raft_consensus.cc:2258] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:41:54.173890 19586 tablet_replica.cc:330] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: stopping tablet replica
I20260619 06:41:54.174305 19586 raft_consensus.cc:2229] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:41:54.174368 19584 ts_tablet_manager.cc:1905] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:41:54.175132 19586 raft_consensus.cc:2258] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:41:54.176038 19585 tablet_replica.cc:330] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: stopping tablet replica
I20260619 06:41:54.176551 19585 raft_consensus.cc:2229] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:41:54.177165 19585 raft_consensus.cc:2258] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:41:54.177567 19586 ts_tablet_manager.cc:1905] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:41:54.179352 19585 ts_tablet_manager.cc:1905] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 06:41:54 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:41:54.186138 19585 ts_tablet_manager.cc:1918] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:41:54.186340 19585 log.cc:1196] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/8ec38b7d36a7421f834b1aad7e0f863e
I20260619 06:41:54.186913 19585 ts_tablet_manager.cc:1939] T 8ec38b7d36a7421f834b1aad7e0f863e P 3207c453a19b44f99ad28a13a8757fa3: Deleting consensus metadata
I20260619 06:41:54.189131 19039 catalog_manager.cc:4841] TS 3207c453a19b44f99ad28a13a8757fa3 (127.7.146.193:37953): tablet 8ec38b7d36a7421f834b1aad7e0f863e (table test-table [id=455ff64a44f946e0a563d0b12f607d94]) successfully deleted
I20260619 06:41:54.191267 19584 ts_tablet_manager.cc:1918] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:41:54.191354 19586 ts_tablet_manager.cc:1918] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:41:54.191545 19584 log.cc:1196] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/8ec38b7d36a7421f834b1aad7e0f863e
I20260619 06:41:54.191586 19586 log.cc:1196] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/8ec38b7d36a7421f834b1aad7e0f863e
I20260619 06:41:54.192102 19584 ts_tablet_manager.cc:1939] T 8ec38b7d36a7421f834b1aad7e0f863e P 2aebef1fc72e4210b975f7c3851f4247: Deleting consensus metadata
I20260619 06:41:54.192127 19586 ts_tablet_manager.cc:1939] T 8ec38b7d36a7421f834b1aad7e0f863e P 9a02c16b5dff4dc8a5101c62a9bb4e9a: Deleting consensus metadata
I20260619 06:41:54.193220 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:54.160188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.7.146.193:37953)
0619 06:41:54.160384 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:54.160389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:54.173367 (+ 12978us) server_negotiation.cc:407] Connection header received
0619 06:41:54.174230 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:54.174237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:54.174330 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:54.174451 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:54.176392 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.177520 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.179385 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.180015 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.180110 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:54.185578 (+ 5468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:54.185593 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:54.185612 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:54.185658 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:54.189792 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:54.190498 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:54.190507 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:54.190510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:54.190663 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:54.191122 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:54.191127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:54.191130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:54.191603 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:54.191910 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:54.192606 (+ 696us) server_negotiation.cc:299] Negotiation successful
0619 06:41:54.192853 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:41:54.193861 19039 catalog_manager.cc:4841] TS 9a02c16b5dff4dc8a5101c62a9bb4e9a (127.7.146.194:40955): tablet 8ec38b7d36a7421f834b1aad7e0f863e (table test-table [id=455ff64a44f946e0a563d0b12f607d94]) successfully deleted
I20260619 06:41:54.194157 19039 catalog_manager.cc:4841] TS 2aebef1fc72e4210b975f7c3851f4247 (127.7.146.195:39941): tablet 8ec38b7d36a7421f834b1aad7e0f863e (table test-table [id=455ff64a44f946e0a563d0b12f607d94]) successfully deleted
I20260619 06:41:54.194748 19260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:54.299376 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 19132
I20260619 06:41:54.320107 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 19271
Jun 19 06:41:54 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:41:54.336833 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:54.320896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.7.146.194:40955)
0619 06:41:54.321089 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:54.321095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:54.321117 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:41:54.321249 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:54.321255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:54.321335 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:54.321447 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:54.322742 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.323932 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.325416 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.326085 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.326212 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:54.330515 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:54.330534 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:54.330555 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:54.330615 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:54.333890 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:54.334812 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:54.334820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:54.334822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:54.334986 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:54.335386 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:54.335390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:54.335392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:54.335739 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:54.335986 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:54.336299 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 06:41:54.336481 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:41:54.337532 19403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:54.449541 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 19271
I20260619 06:41:54.463213 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 19414
Jun 19 06:41:54 dist-test-slave-qfk4 krb5kdc[18576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:41:54.479034 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:41:54.463924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.7.146.195:39941)
0619 06:41:54.464146 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 06:41:54.464153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:41:54.464338 (+ 185us) server_negotiation.cc:407] Connection header received
0619 06:41:54.464614 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:41:54.464618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:41:54.464688 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:41:54.464780 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:41:54.466104 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.466830 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.468243 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:41:54.468733 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:41:54.468857 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:41:54.473434 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:41:54.473455 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:41:54.473475 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:41:54.473527 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:41:54.476256 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:54.477007 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:54.477015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:54.477017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:54.477154 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:41:54.477526 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:41:54.477536 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:41:54.477538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:41:54.477912 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:41:54.478182 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 06:41:54.478504 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 06:41:54.478711 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:41:54.479650 19537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:54.568753 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 19414
I20260619 06:41:54.580853 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 19020
I20260619 06:41:54.582069 19082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:41:54.694612 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 19020
I20260619 06:41:54.711990 7755 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 06:41:55.047494 7755 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 06:41:55.047710 7755 mini_ranger.cc:66] Stopping Ranger...
I20260619 06:41:55.383003 7755 mini_ranger.cc:68] Stopped Ranger
2026-06-19 06:41:55.383 UTC [18606] LOG: received smart shutdown request
2026-06-19 06:41:55.386 UTC [18606] LOG: background worker "logical replication launcher" (PID 18615) exited with exit code 1
2026-06-19 06:41:55.386 UTC [18610] LOG: shutting down
2026-06-19 06:41:55.546 UTC [18606] LOG: database system is shut down
2026-06-19T06:41:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47931 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:41:55 dist-test-slave-qfk4 krb5kdc[19600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:41:55 dist-test-slave-qfk4 krb5kdc[19600](info): set up 2 sockets
Jun 19 06:41:55 dist-test-slave-qfk4 krb5kdc[19600](info): commencing operation
krb5kdc: starting...
W20260619 06:41:56.668841 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:41:56 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:41:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.146.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.146.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/rangeradmin_127.7.146.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/rangeradmin_127.7.146.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.146.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.146.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/rangerlookup_127.7.146.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/rangerlookup_127.7.146.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.146.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.146.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/HTTP_127.7.146.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/HTTP_127.7.146.212@KRBTEST.COM.keytab.
I20260619 06:41:56.970698 7755 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjzr1uN/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/postgres -l logfile start
2026-06-19 06:41:59.293 UTC [19630] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:41:59.293 UTC [19630] LOG: listening on IPv4 address "127.7.146.212", port 52269
2026-06-19 06:41:59.297 UTC [19630] LOG: listening on Unix socket "/tmp/.s.PGSQL.52269"
2026-06-19 06:41:59.316 UTC [19633] LOG: database system was shut down at 2026-06-19 06:41:57 UTC
2026-06-19 06:41:59.319 UTC [19630] LOG: database system is ready to accept connections
I20260619 06:42:00.212436 7755 mini_postgres.cc:96] Postgres bound to 52269
2026-06-19 06:42:00.231 UTC [19641] FATAL: database "slave" does not exist
127.7.146.212:52269 - accepting connections
I20260619 06:42:00.232383 7755 mini_ranger.cc:161] Starting Ranger...
I20260619 06:42:00.256994 7755 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:42:00.714668 7755 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:42:00.714771 7755 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-admin
2026-06-19 06:42:00,982 [I] DB FLAVOR :POSTGRES
2026-06-19 06:42:00,982 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:42:00,982 [I] Checking connection..
2026-06-19 06:42:00,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:42:01,218 [I] Checking connection passed.
2026-06-19 06:42:01,218 [I] --------- Verifying version history table ---------
2026-06-19 06:42:01,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:42:01,476 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:42:01,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:42:01,741 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:42:01,741 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:42:01,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:42:01.989 UTC [19705] WARNING: there is no transaction in progress
2026-06-19 06:42:01,995 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:42:01,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:42:02,248 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:42:02,248 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:42:02,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:42:02,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:42:02,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qfk4.c.gcp-upstream.internal','N') ;"
2026-06-19 06:42:02,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:42:03,155 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:42:03,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:42:03,410 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:42:03,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:42:03,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:42:03,905 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:42:03,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:42:04.817 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:04.832 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:04.848 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:04.863 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:04.879 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:04.894 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:04.956 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:04.963 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:04.977 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:04.987 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:04.997 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:05.011 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:05.021 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:05.028 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:05.035 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:05.042 UTC [19813] WARNING: there is no transaction in progress
2026-06-19 06:42:05,678 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:42:05,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:42:05,925 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:42:05,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:42:06,161 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:42:06,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:42:06,388 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:42:06,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:42:06,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:42:06,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qfk4.c.gcp-upstream.internal';"
2026-06-19 06:42:07,120 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:42:07,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:42:07,378 [I] DB_PATCHES have already been applied
I20260619 06:42:07.384375 7755 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjzr1uN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:42:07.384464 7755 mini_ranger.cc:193] Using host: 127.7.146.212
I20260619 06:42:07.397704 7755 mini_ranger.cc:227] Ranger admin URL: http://127.7.146.212:49967
Jun 19, 2026 6:42:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:42:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:42:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:42:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:42:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:42:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:42:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:42:08.438560 7755 mini_ranger.cc:160] Time spent starting Ranger: real 8.206s user 0.005s sys 0.032s
Jun 19, 2026 6:42:28 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:42:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:42:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:42:28 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:42:29 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 06:42:30.113242 7755 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.146.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.146.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/rangerkms_127.7.146.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.146.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/rangerkms_127.7.146.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.146.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.146.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.146.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/HTTP_127.7.146.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.146.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/HTTP_127.7.146.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 06:42:30 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 06:42:30.262835 7755 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 06:42:30.281065 7755 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 06:42:31.107492 7755 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 06:42:31.107587 7755 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-kms
I20260619 06:42:31.107625 7755 mini_ranger_kms.cc:227] Using postgres at 127.7.146.212:52269
2026-06-19 06:42:31,420 [I] DB FLAVOR :POSTGRES
2026-06-19 06:42:31,420 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:42:31,420 [I] Checking connection
2026-06-19 06:42:31,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 06:42:31,674 [I] connection success
2026-06-19 06:42:31,675 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 06:42:31,675 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 06:42:31,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 06:42:31,925 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 06:42:31,926 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:42:31,926 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 06:42:31,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.146.212:52269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 06:42:32,178 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 06:42:32.643112 7755 mini_ranger_kms.cc:310] Created kms service
I20260619 06:42:32.843863 7755 mini_ranger_kms.cc:326] Created kudu user
I20260619 06:42:32.887825 7755 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 06:42:32 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:33.287369 7755 mini_ranger_kms.cc:384] Added ranger policy
I20260619 06:42:33.287703 7755 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjzr1uN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjzr1uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 06:42:33.287765 7755 mini_ranger_kms.cc:240] Using host: 127.7.146.212
I20260619 06:42:33.301440 7755 mini_ranger_kms.cc:276] Ranger KMS PID: 19993
I20260619 06:42:33.301565 7755 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.146.212:37165
Jun 19, 2026 6:42:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:42:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:42:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:42:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:42:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:42:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 06:42:35.414529 7755 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.152s user 0.064s sys 0.262s
I20260619 06:42:35.414733 7755 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 06:42:35.414888 7755 mini_ranger_kms.cc:401] 127.7.146.212:37165/kms/v1/keys
Jun 19 06:42:35 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.146.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 06:42:36 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.146.212@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
Jun 19 06:42:36 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.146.212@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
Jun 19 06:42:38 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:42:38.783943 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38963
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:38963
--ranger_config_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:37165/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:42:39.103475 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:39.103790 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:39.103919 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:39.111161 20044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:42:39.111297 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:39.111373 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:39.111444 20044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:42:39.111505 20044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:42:39.121752 20044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.146.212:37165/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:38963
--ranger_config_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:39.123075 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:39.124406 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:39.135560 20049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:39.135561 20050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:39.135676 20052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:39.136318 20044 server_base.cc:1034] running on GCE node
I20260619 06:42:39.137115 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:39.138489 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:39.139703 20044 hybrid_clock.cc:648] HybridClock initialized: now 1781851359139653 us; error 45 us; skew 500 ppm
I20260619 06:42:39.140058 20044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:39 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851359, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:39.143020 20044 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:42:39.144379 20044 webserver.cc:460] Webserver started at http://127.7.146.254:36663/ using document root <none> and password file <none>
I20260619 06:42:39.145056 20044 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:39.145169 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:39.145452 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:42:39 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851359, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:39.197175 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "ec36bb821331475bba28aaec20f3a153"
format_stamp: "Formatted at 2026-06-19 06:42:39 on dist-test-slave-qfk4"
server_key: "2d165e65c5897512d18e6a6e37e6c59b"
server_key_iv: "1e263d19f6bf5346012e4c53ef6a5982"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:39.198212 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "ec36bb821331475bba28aaec20f3a153"
format_stamp: "Formatted at 2026-06-19 06:42:39 on dist-test-slave-qfk4"
server_key: "2d165e65c5897512d18e6a6e37e6c59b"
server_key_iv: "1e263d19f6bf5346012e4c53ef6a5982"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:39.203764 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:42:39.224617 20059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:39.226119 20044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:42:39.226277 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "ec36bb821331475bba28aaec20f3a153"
format_stamp: "Formatted at 2026-06-19 06:42:39 on dist-test-slave-qfk4"
server_key: "2d165e65c5897512d18e6a6e37e6c59b"
server_key_iv: "1e263d19f6bf5346012e4c53ef6a5982"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:39.226492 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:39.239850 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:39.244047 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:39.244352 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:39.262346 20044 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:38963
I20260619 06:42:39.262379 20110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:38963 every 8 connection(s)
I20260619 06:42:39.263731 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:42:39.267503 20111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:42:39.267982 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 20044
I20260619 06:42:39.268216 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:42:39.269052 7755 external_mini_cluster.cc:1428] Setting key 073c744fefa35f38fba440441dccefb1
I20260619 06:42:39.282322 20111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153: Bootstrap starting.
I20260619 06:42:39.286171 20111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:39.287358 20111 log.cc:824] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:39.290838 20111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153: No bootstrap required, opened a new log
Jun 19 06:42:39 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:39.300546 20111 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } }
I20260619 06:42:39.301047 20111 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:39.301143 20111 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec36bb821331475bba28aaec20f3a153, State: Initialized, Role: FOLLOWER
I20260619 06:42:39.301829 20111 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } }
I20260619 06:42:39.302085 20111 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:42:39.302251 20111 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:42:39.302456 20111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:39.305742 20114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:39.271209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.7.146.254:38963)
0619 06:42:39.281129 (+ 9920us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:39.281146 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:39.281591 (+ 445us) server_negotiation.cc:407] Connection header received
0619 06:42:39.283989 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:39.284026 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:39.284630 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:39.285072 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:39.286651 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:39.288428 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:39.290451 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:39.291532 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:39.291728 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:39.296789 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:39.296800 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:39.296821 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:39.296879 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:39.300789 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:39.301630 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:39.301639 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:39.301648 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:39.301830 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:39.302269 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:39.302274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:39.302277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:39.303016 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:39.303343 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:39.303769 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 06:42:39.304314 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9666,"thread_start_us":9440,"threads_started":1}
I20260619 06:42:39.305725 20111 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } }
I20260619 06:42:39.306299 20111 leader_election.cc:302] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec36bb821331475bba28aaec20f3a153; no voters:
I20260619 06:42:39.306951 20111 leader_election.cc:288] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:42:39.307087 20116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:42:39.311046 20111 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:42:39.313936 20116 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 1 LEADER]: Becoming Leader. State: Replica: ec36bb821331475bba28aaec20f3a153, State: Running, Role: LEADER
I20260619 06:42:39.314476 20116 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } }
I20260619 06:42:39.327745 20117 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec36bb821331475bba28aaec20f3a153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } } }
I20260619 06:42:39.327718 20118 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec36bb821331475bba28aaec20f3a153. Latest consensus state: current_term: 1 leader_uuid: "ec36bb821331475bba28aaec20f3a153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } } }
I20260619 06:42:39.328244 20117 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:39.328245 20118 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:39.341631 20128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:42:39.342418 20111 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:42:39.353659 20128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:42:39.373195 20128 catalog_manager.cc:1347] Generated new cluster ID: 9f4c2fa94d4e4a50bf9c7c2f6579dd01
I20260619 06:42:39.373378 20128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:42:39.388681 20128 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:42:39.390691 20128 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:42:39.403131 20128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153: Generated new TSK 0
I20260619 06:42:39.404263 20128 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:42:40.215773 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:37165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38963
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:42:40 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851360, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 06:42:40.651873 20156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:40.652287 20156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:40.652443 20156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:40.664022 20156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:40.664240 20156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:40.664459 20156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:42:40.676352 20156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.146.212:37165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:40.677704 20156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:40.678980 20156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:40.698838 20169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:40.700891 20167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:40.701571 20166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:40.702953 20156 server_base.cc:1034] running on GCE node
I20260619 06:42:40.703784 20156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:40.705075 20156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:40.706377 20156 hybrid_clock.cc:648] HybridClock initialized: now 1781851360706310 us; error 53 us; skew 500 ppm
I20260619 06:42:40.706766 20156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:40 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851360, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:40.710839 20156 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:42:40.718374 20156 webserver.cc:460] Webserver started at http://127.7.146.193:34837/ using document root <none> and password file <none>
I20260619 06:42:40.719352 20156 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:40.719511 20156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:40.719882 20156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:42:40 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851360, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:40.741221 20156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "451aa0dfc8a84be185b5e918704229ed"
format_stamp: "Formatted at 2026-06-19 06:42:40 on dist-test-slave-qfk4"
server_key: "8bfe175589597e8648b2be0eb8b1f5b9"
server_key_iv: "57bc300f0620ae220291b9c827bddc52"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:40.742445 20156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "451aa0dfc8a84be185b5e918704229ed"
format_stamp: "Formatted at 2026-06-19 06:42:40 on dist-test-slave-qfk4"
server_key: "8bfe175589597e8648b2be0eb8b1f5b9"
server_key_iv: "57bc300f0620ae220291b9c827bddc52"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:40.756609 20156 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.007s sys 0.004s
I20260619 06:42:40.789312 20176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:40.793390 20156 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.011s sys 0.005s
I20260619 06:42:40.793591 20156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "451aa0dfc8a84be185b5e918704229ed"
format_stamp: "Formatted at 2026-06-19 06:42:40 on dist-test-slave-qfk4"
server_key: "8bfe175589597e8648b2be0eb8b1f5b9"
server_key_iv: "57bc300f0620ae220291b9c827bddc52"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:40.793812 20156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:40.851706 20156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:40.857414 20156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:40.857820 20156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:40.859393 20156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:40.861153 20156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:40.861250 20156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:40.861366 20156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:40.861421 20156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:42:40 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851360, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:41.002161 20156 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:32817
I20260619 06:42:41.002211 20288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:32817 every 8 connection(s)
I20260619 06:42:41.003887 20156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:42:41.011731 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 20156
I20260619 06:42:41.011914 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:42:41.012732 7755 external_mini_cluster.cc:1428] Setting key a1d43d7fa37354ac62989424929bdf93
Jun 19 06:42:41 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851360, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
I20260619 06:42:41.043488 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:41.008134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:49707 (local address 127.7.146.254:38963)
0619 06:42:41.023273 (+ 15139us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:41.023280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:41.023305 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:42:41.023372 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:41.023377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:41.023457 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:41.023585 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:41.026895 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:41.028059 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:41.030685 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:41.031425 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:41.031544 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:41.035944 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:41.035950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:41.035955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:41.036017 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:41.039439 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:41.040393 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:41.040399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:41.040402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:41.040550 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:41.041095 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:41.041100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:41.041104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:41.041480 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:41.041609 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:41.042806 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0619 06:42:41.043055 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15002,"thread_start_us":15540,"threads_started":1}
I20260619 06:42:41.044407 20291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:41.007536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38963 (local address 127.7.146.193:49707)
0619 06:42:41.008294 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:41.008342 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:41.010709 (+ 2367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:41.025108 (+ 14399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:41.025122 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:41.025772 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:41.026654 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:41.026680 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:41.028322 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:41.028328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:41.030496 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:41.030522 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:41.031846 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:41.031853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:41.032146 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:41.032938 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:41.032968 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:41.035672 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:41.039613 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:41.039623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:41.039643 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:41.040247 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:41.040709 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:41.040714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:41.040718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:41.040956 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:41.041688 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:41.041697 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:41.042145 (+ 448us) client_negotiation.cc:769] Sending connection context
0619 06:42:41.042425 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 06:42:41.042949 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":8076,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
I20260619 06:42:41.046409 20289 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38963
I20260619 06:42:41.046733 20289 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:41.047544 20289 heartbeater.cc:511] Master 127.7.146.254:38963 requested a full tablet report, sending...
I20260619 06:42:41.050268 20076 ts_manager.cc:194] Registered new tserver with Master: 451aa0dfc8a84be185b5e918704229ed (127.7.146.193:32817)
I20260619 06:42:41.052350 20076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:49707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:42:41.079854 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:37165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38963
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:42:41 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851360, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
W20260619 06:42:41.524813 20295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:41.525177 20295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:41.525300 20295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:41.534341 20295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:41.534477 20295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:41.534634 20295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:42:41.544646 20295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.146.212:37165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:41.545979 20295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:41.547397 20295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:41.559856 20309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:41.560061 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:41.559895 20310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:41.560838 20295 server_base.cc:1034] running on GCE node
I20260619 06:42:41.561395 20295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:41.562294 20295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:41.563499 20295 hybrid_clock.cc:648] HybridClock initialized: now 1781851361563434 us; error 47 us; skew 500 ppm
I20260619 06:42:41.563792 20295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:41 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851361, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:41.566876 20295 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:42:41.568270 20295 webserver.cc:460] Webserver started at http://127.7.146.194:41123/ using document root <none> and password file <none>
I20260619 06:42:41.568989 20295 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:41.569103 20295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:41.569414 20295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:42:41 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851361, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:41.585552 20295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "02f88f0ea0e040cd814178ef0bcdffa7"
format_stamp: "Formatted at 2026-06-19 06:42:41 on dist-test-slave-qfk4"
server_key: "b14b833f28b055f71ea2f628e703a601"
server_key_iv: "4c529e3ea5ef7c2170f9a5c86caf2440"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:41.586392 20295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "02f88f0ea0e040cd814178ef0bcdffa7"
format_stamp: "Formatted at 2026-06-19 06:42:41 on dist-test-slave-qfk4"
server_key: "b14b833f28b055f71ea2f628e703a601"
server_key_iv: "4c529e3ea5ef7c2170f9a5c86caf2440"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:41.591773 20295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 06:42:41.608387 20319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:41.610175 20295 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:42:41.610342 20295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "02f88f0ea0e040cd814178ef0bcdffa7"
format_stamp: "Formatted at 2026-06-19 06:42:41 on dist-test-slave-qfk4"
server_key: "b14b833f28b055f71ea2f628e703a601"
server_key_iv: "4c529e3ea5ef7c2170f9a5c86caf2440"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:41.610553 20295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:41.632400 20295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:41.636178 20295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:41.636448 20295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:41.637405 20295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:41.638768 20295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:41.638845 20295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:41.638957 20295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:41.639004 20295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:41.670181 20295 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:41063
I20260619 06:42:41.670219 20431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:41063 every 8 connection(s)
I20260619 06:42:41.671409 20295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:42:41.681555 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 20295
I20260619 06:42:41.681744 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:42:41.682206 7755 external_mini_cluster.cc:1428] Setting key 9b61a915029a7fdd3488dc02cd298c2b
Jun 19 06:42:41 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851361, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:41.690897 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:41.674049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:55475 (local address 127.7.146.254:38963)
0619 06:42:41.674611 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:41.674617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:41.675496 (+ 879us) server_negotiation.cc:407] Connection header received
0619 06:42:41.677439 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:41.677443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:41.677522 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:41.677623 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:41.679142 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:41.679857 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:41.681663 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:41.682090 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:41.682200 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:41.685622 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:41.685627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:41.685630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:41.685663 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:41.687985 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:41.688689 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:41.688693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:41.688696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:41.688812 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:41.689236 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:41.689239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:41.689241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:41.689510 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:41.689640 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:41.690324 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 06:42:41.690548 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":264,"threads_started":1}
I20260619 06:42:41.691838 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:41.674543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38963 (local address 127.7.146.194:55475)
0619 06:42:41.675287 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:41.675328 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:41.677165 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:41.677886 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:41.677897 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:41.678328 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:41.678989 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:41.679007 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:41.680010 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:41.680014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:41.681453 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:41.681542 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:41.682334 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:41.682343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:41.682647 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:41.683325 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:41.683348 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:41.685415 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:41.688132 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:41.688141 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:41.688156 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:41.688550 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:41.688913 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:41.688918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:41.688922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:41.689124 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:41.689636 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:41.689643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:41.689961 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 06:42:41.690191 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 06:42:41.690710 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":275,"threads_started":1}
I20260619 06:42:41.693239 20432 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38963
I20260619 06:42:41.693558 20432 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:41.694360 20432 heartbeater.cc:511] Master 127.7.146.254:38963 requested a full tablet report, sending...
I20260619 06:42:41.695900 20076 ts_manager.cc:194] Registered new tserver with Master: 02f88f0ea0e040cd814178ef0bcdffa7 (127.7.146.194:41063)
I20260619 06:42:41.696722 20076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:55475
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:42:41.729389 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:37165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38963
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:42:42.041329 20438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:42.041642 20438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:42.041778 20438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:42.048998 20438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:42.049124 20438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:42.049266 20438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:42:42.056449 20289 heartbeater.cc:503] Master 127.7.146.254:38963 was elected leader, sending a full tablet report...
I20260619 06:42:42.059243 20438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.146.212:37165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:38963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:42.060546 20438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:42.061833 20438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:42.074316 20443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:42.074276 20446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:42.074347 20444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:42.075194 20438 server_base.cc:1034] running on GCE node
I20260619 06:42:42.075747 20438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:42.076715 20438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:42.077931 20438 hybrid_clock.cc:648] HybridClock initialized: now 1781851362077873 us; error 48 us; skew 500 ppm
I20260619 06:42:42.078255 20438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:42 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851362, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:42.081326 20438 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:42:42.082705 20438 webserver.cc:460] Webserver started at http://127.7.146.195:34555/ using document root <none> and password file <none>
I20260619 06:42:42.083353 20438 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:42.083462 20438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:42.083745 20438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:42:42 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851362, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:42.097270 20438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "c4a5bea8ce6444e9874a6d50d863db66"
format_stamp: "Formatted at 2026-06-19 06:42:42 on dist-test-slave-qfk4"
server_key: "a5eb97033adb95aef1db61c5331e76f1"
server_key_iv: "be3e6a447ca4fe428e4aa7174ceb2c43"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:42.098174 20438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "c4a5bea8ce6444e9874a6d50d863db66"
format_stamp: "Formatted at 2026-06-19 06:42:42 on dist-test-slave-qfk4"
server_key: "a5eb97033adb95aef1db61c5331e76f1"
server_key_iv: "be3e6a447ca4fe428e4aa7174ceb2c43"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:42.103591 20438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:42:42.120715 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:42.122429 20438 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:42:42.122593 20438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "c4a5bea8ce6444e9874a6d50d863db66"
format_stamp: "Formatted at 2026-06-19 06:42:42 on dist-test-slave-qfk4"
server_key: "a5eb97033adb95aef1db61c5331e76f1"
server_key_iv: "be3e6a447ca4fe428e4aa7174ceb2c43"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:42.122803 20438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:42.143172 20438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:42.146953 20438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:42.147228 20438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:42.148164 20438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:42.149531 20438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:42.149600 20438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:42.149708 20438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:42.149753 20438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:42.182772 20438 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:35559
I20260619 06:42:42.182804 20565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:35559 every 8 connection(s)
I20260619 06:42:42.184000 20438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:42:42.189653 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 20438
I20260619 06:42:42.189837 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:42:42.190323 7755 external_mini_cluster.cc:1428] Setting key 8fc1bd2910f1bf84dbf14bef19345cdb
Jun 19 06:42:42 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851362, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:42.203230 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:42.186500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:50231 (local address 127.7.146.254:38963)
0619 06:42:42.186662 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:42.186666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:42.187759 (+ 1093us) server_negotiation.cc:407] Connection header received
0619 06:42:42.189601 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:42.189607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:42.189728 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:42.189845 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:42.191426 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:42.192411 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:42.194210 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:42.194698 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:42.194794 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:42.198118 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:42.198123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:42.198127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:42.198178 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:42.200672 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:42.201362 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:42.201366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:42.201368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:42.201485 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:42.201841 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:42.201844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:42.201846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:42.202118 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:42.202228 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:42.202812 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 06:42:42.202977 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:42:42.204216 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:42.186882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38963 (local address 127.7.146.195:50231)
0619 06:42:42.187592 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:42.187630 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:42.189313 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:42.190081 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:42.190090 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:42.190515 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:42.191263 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:42.191289 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:42.192587 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:42.192592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:42.194076 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:42.194095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:42.194873 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:42.194878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:42.195105 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:42.195730 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:42.195753 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:42.197868 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:42.200806 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:42.200813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:42.200828 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:42.201253 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:42.201573 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:42.201576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:42.201579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:42.201760 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:42.202207 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:42.202212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:42.202518 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 06:42:42.202730 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 06:42:42.203090 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":278,"threads_started":1}
I20260619 06:42:42.205523 20566 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38963
I20260619 06:42:42.205833 20566 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:42.206508 20566 heartbeater.cc:511] Master 127.7.146.254:38963 requested a full tablet report, sending...
I20260619 06:42:42.207834 20076 ts_manager.cc:194] Registered new tserver with Master: c4a5bea8ce6444e9874a6d50d863db66 (127.7.146.195:35559)
I20260619 06:42:42.208673 20076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:50231
I20260619 06:42:42.218140 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:42:42.219764 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 20156
Jun 19 06:42:42 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:42:42.235822 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:42.220770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.7.146.193:32817)
0619 06:42:42.221341 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:42.221348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:42.221373 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:42:42.221434 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:42.221441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:42.221659 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:42.221780 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:42.223189 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:42.224155 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:42.225637 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:42.226170 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:42.226281 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:42.230393 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:42.230404 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:42.230420 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:42.230457 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:42.233150 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:42.233806 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:42.233814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:42.233816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:42.233959 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:42.234306 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:42.234309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:42.234311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:42.234646 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:42.234922 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:42.235214 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:42:42.235474 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":271,"threads_started":1}
I20260619 06:42:42.237252 20284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:42.325059 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 20156
I20260619 06:42:42.337524 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 20295
Jun 19 06:42:42 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:42:42.353209 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:42.338421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.7.146.194:41063)
0619 06:42:42.339004 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:42.339011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:42.339037 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:42:42.339095 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:42.339101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:42.339318 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:42.339451 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:42.340665 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:42.341672 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:42.342944 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:42.343479 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:42.343574 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:42.347715 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:42.347727 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:42.347740 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:42.347774 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:42.350502 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:42.351176 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:42.351183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:42.351186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:42.351313 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:42.351662 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:42.351666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:42.351668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:42.351989 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:42.352279 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:42.352618 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 06:42:42.352882 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":332,"threads_started":1}
I20260619 06:42:42.354439 20427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:42.435712 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 20295
I20260619 06:42:42.447211 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 20438
Jun 19 06:42:42 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:42:42.463256 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:42.448206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.7.146.195:35559)
0619 06:42:42.448896 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:42.448902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:42.448931 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:42:42.448992 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:42.448998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:42.449214 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:42.449360 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:42.450747 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:42.451756 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:42.453088 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:42.453648 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:42.453757 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:42.457992 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:42.458004 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:42.458019 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:42.458052 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:42.460513 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:42.461298 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:42.461305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:42.461307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:42.461437 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:42.461780 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:42.461784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:42.461786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:42.462108 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:42.462414 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:42.462688 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 06:42:42.462927 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":390,"threads_started":1}
I20260619 06:42:42.464491 20561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:42.545681 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 20438
I20260619 06:42:42.557448 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 20044
I20260619 06:42:42.558758 20106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:42.661934 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 20044
I20260619 06:42:42.679050 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38963
--webserver_interface=127.7.146.254
--webserver_port=36663
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:38963
--ranger_config_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:37165/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.146.212:37165/kms with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:42:42.992197 20577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:42.992491 20577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:42.992637 20577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:43.000003 20577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:42:43.000131 20577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:43.000222 20577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:43.000303 20577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:42:43.000371 20577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:42:43.010668 20577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.146.212:37165/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:38963
--ranger_config_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:43.011920 20577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:43.013252 20577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:43.024636 20585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:43.024634 20583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:43.024652 20582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:43.025262 20577 server_base.cc:1034] running on GCE node
I20260619 06:42:43.026050 20577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:43.027546 20577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:43.028767 20577 hybrid_clock.cc:648] HybridClock initialized: now 1781851363028708 us; error 52 us; skew 500 ppm
I20260619 06:42:43.029125 20577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:43 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:43.032322 20577 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:42:43.033830 20577 webserver.cc:460] Webserver started at http://127.7.146.254:36663/ using document root <none> and password file <none>
I20260619 06:42:43.034513 20577 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:43.034644 20577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:42:43 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:43.084233 20577 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 06:42:43.087800 20593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:43.089227 20577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:42:43.089370 20577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "ec36bb821331475bba28aaec20f3a153"
format_stamp: "Formatted at 2026-06-19 06:42:39 on dist-test-slave-qfk4"
server_key: "2d165e65c5897512d18e6a6e37e6c59b"
server_key_iv: "1e263d19f6bf5346012e4c53ef6a5982"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:43.090281 20577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:43.136638 20577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:43.140717 20577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:43.140956 20577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:43.157763 20577 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:38963
I20260619 06:42:43.157773 20644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:38963 every 8 connection(s)
I20260619 06:42:43.159266 20577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:42:43.160765 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 20577
I20260619 06:42:43.161772 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:37165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:32817
--local_ip_for_outbound_sockets=127.7.146.193
--tserver_master_addrs=127.7.146.254:38963
--webserver_port=34837
--webserver_interface=127.7.146.193
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.146.212:37165/kms with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:42:43.168418 20645 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:42:43.171993 20645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153: Bootstrap starting.
I20260619 06:42:43.190595 20645 log.cc:824] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:43.199361 20645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:42:43.199854 20645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153: Bootstrap complete.
I20260619 06:42:43.208530 20645 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } }
I20260619 06:42:43.209408 20645 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec36bb821331475bba28aaec20f3a153, State: Initialized, Role: FOLLOWER
I20260619 06:42:43.210017 20645 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } }
I20260619 06:42:43.210230 20645 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:42:43.210337 20645 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:42:43.210485 20645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:42:43.214181 20645 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } }
I20260619 06:42:43.214649 20645 leader_election.cc:302] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec36bb821331475bba28aaec20f3a153; no voters:
I20260619 06:42:43.215226 20645 leader_election.cc:288] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:42:43.215361 20649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:42:43.216410 20649 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [term 2 LEADER]: Becoming Leader. State: Replica: ec36bb821331475bba28aaec20f3a153, State: Running, Role: LEADER
I20260619 06:42:43.216964 20649 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } }
I20260619 06:42:43.217303 20645 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:42:43.219377 20650 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ec36bb821331475bba28aaec20f3a153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } } }
I20260619 06:42:43.219751 20650 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:43.223459 20651 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec36bb821331475bba28aaec20f3a153. Latest consensus state: current_term: 2 leader_uuid: "ec36bb821331475bba28aaec20f3a153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec36bb821331475bba28aaec20f3a153" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38963 } } }
I20260619 06:42:43.223778 20651 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:43.225286 20654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:42:43.231101 20654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:42:43.232515 20654 catalog_manager.cc:1259] Loaded cluster ID: 9f4c2fa94d4e4a50bf9c7c2f6579dd01
I20260619 06:42:43.232642 20654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:42:43.236755 20654 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:42:43.238281 20654 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153: Loaded TSK: 0
I20260619 06:42:43.239871 20654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:42:43.240994 20645 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 06:42:43.600616 20647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:43.601039 20647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:43.601246 20647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:43.612668 20647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:43.612879 20647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:43.613128 20647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:42:43.628798 20647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.146.212:37165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:32817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=34837
--tserver_master_addrs=127.7.146.254:38963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:43.630858 20647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:43.632777 20647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:43.681322 20682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:43.689079 20683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:43.692634 20685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:43.692948 20647 server_base.cc:1034] running on GCE node
I20260619 06:42:43.694890 20647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:43.704866 20647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:43.708371 20647 hybrid_clock.cc:648] HybridClock initialized: now 1781851363708232 us; error 136 us; skew 500 ppm
I20260619 06:42:43.708951 20647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:43 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:43.713274 20647 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:42:43.741003 20647 webserver.cc:460] Webserver started at http://127.7.146.193:34837/ using document root <none> and password file <none>
I20260619 06:42:43.742182 20647 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:43.742365 20647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:42:43 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:43.786623 20647 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.003s sys 0.005s
I20260619 06:42:43.790503 20694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:43.792299 20647 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 06:42:43.792516 20647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "451aa0dfc8a84be185b5e918704229ed"
format_stamp: "Formatted at 2026-06-19 06:42:40 on dist-test-slave-qfk4"
server_key: "8bfe175589597e8648b2be0eb8b1f5b9"
server_key_iv: "57bc300f0620ae220291b9c827bddc52"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:43.793888 20647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:43.813201 20647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:43.818215 20647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:43.818547 20647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:43.819775 20647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:43.821573 20647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:43.821681 20647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:43.821780 20647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:43.821847 20647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:43.856902 20647 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:32817
I20260619 06:42:43.856971 20806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:32817 every 8 connection(s)
I20260619 06:42:43.858860 20647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:42:43.859855 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 20647
I20260619 06:42:43.860670 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:37165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:41063
--local_ip_for_outbound_sockets=127.7.146.194
--tserver_master_addrs=127.7.146.254:38963
--webserver_port=41123
--webserver_interface=127.7.146.194
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.146.212:37165/kms with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:42:43 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:43.910249 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:43.868538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38963 (local address 127.7.146.193:35075)
0619 06:42:43.877321 (+ 8783us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:43.877392 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:43.880190 (+ 2798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:43.882973 (+ 2783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:43.882986 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:43.883694 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:43.884810 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:43.884840 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:43.887086 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:43.887091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:43.889213 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:43.889229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:43.893613 (+ 4384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:43.893622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:43.893923 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:43.895364 (+ 1441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:43.895398 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:43.898225 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:43.905127 (+ 6902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:43.905137 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:43.905156 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:43.905754 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:43.906296 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:43.906301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:43.906305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:43.906558 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:43.907576 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:43.907585 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:43.907991 (+ 406us) client_negotiation.cc:769] Sending connection context
0619 06:42:43.908278 (+ 287us) client_negotiation.cc:240] Negotiation successful
0619 06:42:43.908960 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7144,"thread_start_us":6880,"threads_started":1}
I20260619 06:42:43.912550 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:43.869421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:35075 (local address 127.7.146.254:38963)
0619 06:42:43.881177 (+ 11756us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:43.881199 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:43.881254 (+ 55us) server_negotiation.cc:407] Connection header received
0619 06:42:43.881744 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:43.881767 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:43.882321 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:43.882791 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:43.885167 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:43.886893 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:43.892066 (+ 5173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:43.893309 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:43.893592 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:43.898524 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:43.898535 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:43.898556 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:43.898612 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:43.904901 (+ 6289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:43.905930 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:43.905941 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:43.905950 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:43.906166 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:43.906682 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:43.906687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:43.906691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:43.907331 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:43.907498 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:43.910927 (+ 3429us) server_negotiation.cc:299] Negotiation successful
0619 06:42:43.911348 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11440,"thread_start_us":11184,"threads_started":1}
I20260619 06:42:43.916666 20807 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38963
I20260619 06:42:43.917004 20807 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:43.917738 20807 heartbeater.cc:511] Master 127.7.146.254:38963 requested a full tablet report, sending...
W20260619 06:42:43.920481 20807 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:38963 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:42:43.920708 20807 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:43.920956 20807 heartbeater.cc:511] Master 127.7.146.254:38963 requested a full tablet report, sending...
I20260619 06:42:43.922077 20807 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:43.922320 20807 heartbeater.cc:511] Master 127.7.146.254:38963 requested a full tablet report, sending...
W20260619 06:42:43.923374 20807 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:42:44.079594 20818 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 06:42:44.287493 20808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:44.287904 20808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:44.288108 20808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:44.299955 20808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:44.300072 20808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:44.300232 20808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:42:44.310802 20808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.146.212:37165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:41063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=41123
--tserver_master_addrs=127.7.146.254:38963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:44.312206 20808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:44.313599 20808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:44.347784 20825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:44.350310 20826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:44.351656 20808 server_base.cc:1034] running on GCE node
W20260619 06:42:44.351926 20830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:44.352559 20808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:44.354084 20808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:44.355345 20808 hybrid_clock.cc:648] HybridClock initialized: now 1781851364355292 us; error 52 us; skew 500 ppm
I20260619 06:42:44.355734 20808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:44 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851364, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:44.362936 20808 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:42:44.382359 20808 webserver.cc:460] Webserver started at http://127.7.146.194:41123/ using document root <none> and password file <none>
I20260619 06:42:44.383373 20808 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:44.383548 20808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:42:44 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851364, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:44.409751 20808 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:42:44.413396 20838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:44.415201 20808 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:42:44.415421 20808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "02f88f0ea0e040cd814178ef0bcdffa7"
format_stamp: "Formatted at 2026-06-19 06:42:41 on dist-test-slave-qfk4"
server_key: "b14b833f28b055f71ea2f628e703a601"
server_key_iv: "4c529e3ea5ef7c2170f9a5c86caf2440"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:44.416880 20808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 19 06:42:44 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851364, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:44.458043 20808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:44.463832 20808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:44.464159 20808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:44.465504 20808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:44.467206 20808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:44.467317 20808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:44.467428 20808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:44.467504 20808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:44.523506 20808 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:41063
I20260619 06:42:44.523537 20951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:41063 every 8 connection(s)
I20260619 06:42:44.525279 20808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:42:44.530464 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 20808
I20260619 06:42:44.531263 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.146.212:37165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:35559
--local_ip_for_outbound_sockets=127.7.146.195
--tserver_master_addrs=127.7.146.254:38963
--webserver_port=34555
--webserver_interface=127.7.146.195
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.146.212:37165/kms with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:42:44 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851364, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:44.559118 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:44.528849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:47751 (local address 127.7.146.254:38963)
0619 06:42:44.540472 (+ 11623us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:44.540479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:44.540499 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:42:44.540589 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:44.540594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:44.540674 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:44.540776 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:44.542890 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:44.543976 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:44.546241 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:44.546856 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:44.546958 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:44.551390 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:44.551397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:44.551402 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:44.551454 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:44.555056 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:44.556106 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:44.556112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:44.556115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:44.556272 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:44.556888 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:44.556893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:44.556896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:44.557270 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:44.557430 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:44.558497 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0619 06:42:44.558740 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11479,"thread_start_us":11710,"threads_started":1}
I20260619 06:42:44.561645 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:44.529545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38963 (local address 127.7.146.194:47751)
0619 06:42:44.537262 (+ 7717us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:44.537316 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:44.539770 (+ 2454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:44.541200 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:44.541214 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:44.541817 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:44.542677 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:44.542704 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:44.544174 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:44.544179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:44.546071 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:44.546086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:44.547326 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:44.547332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:44.547609 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:44.548398 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:44.548436 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:44.551107 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:44.555255 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:44.555266 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:44.555288 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:44.555942 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:44.556443 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:44.556448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:44.556452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:44.556752 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:44.557528 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:44.557538 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:44.558035 (+ 497us) client_negotiation.cc:769] Sending connection context
0619 06:42:44.559600 (+ 1565us) client_negotiation.cc:240] Negotiation successful
0619 06:42:44.560229 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7388,"thread_start_us":7167,"threads_started":1}
I20260619 06:42:44.563581 20952 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38963
I20260619 06:42:44.563913 20952 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:44.564806 20952 heartbeater.cc:511] Master 127.7.146.254:38963 requested a full tablet report, sending...
I20260619 06:42:44.566905 20610 ts_manager.cc:194] Registered new tserver with Master: 02f88f0ea0e040cd814178ef0bcdffa7 (127.7.146.194:41063)
I20260619 06:42:44.569137 20610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:47751
Jun 19 06:42:44 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851364, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:44.923563 20807 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:44.923995 20807 heartbeater.cc:511] Master 127.7.146.254:38963 requested a full tablet report, sending...
I20260619 06:42:44.925302 20610 ts_manager.cc:194] Registered new tserver with Master: 451aa0dfc8a84be185b5e918704229ed (127.7.146.193:32817)
I20260619 06:42:44.926510 20610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:35075
W20260619 06:42:44.983268 20956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:44.983573 20956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:44.983718 20956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:44.991073 20956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:44.991200 20956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:44.991377 20956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:42:45.007194 20956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.146.212:37165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:35559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=34555
--tserver_master_addrs=127.7.146.254:38963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:45.009270 20956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:45.011092 20956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:45.060359 20962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:45.061026 20961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:45.061829 20965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:45.062872 20956 server_base.cc:1034] running on GCE node
I20260619 06:42:45.063791 20956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:45.073038 20956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:45.080678 20956 hybrid_clock.cc:648] HybridClock initialized: now 1781851365080576 us; error 83 us; skew 500 ppm
I20260619 06:42:45.081152 20956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:45 dist-test-slave-qfk4 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851365, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:45.085544 20956 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:42:45.087723 20956 webserver.cc:460] Webserver started at http://127.7.146.195:34555/ using document root <none> and password file <none>
I20260619 06:42:45.088743 20956 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:45.088953 20956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:42:45 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851365, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:45.123159 20956 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.007s sys 0.000s
I20260619 06:42:45.149300 20972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:45.151593 20956 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.009s sys 0.015s
I20260619 06:42:45.151870 20956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "c4a5bea8ce6444e9874a6d50d863db66"
format_stamp: "Formatted at 2026-06-19 06:42:42 on dist-test-slave-qfk4"
server_key: "a5eb97033adb95aef1db61c5331e76f1"
server_key_iv: "be3e6a447ca4fe428e4aa7174ceb2c43"
server_key_version: "kuduclusterkey@0"
I20260619 06:42:45.153424 20956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 19 06:42:45 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851364, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for HTTP/127.7.146.212@KRBTEST.COM
I20260619 06:42:45.228229 20956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:45.233574 20956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:45.233922 20956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:45.270253 20956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:45.277679 20956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:45.277856 20956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 06:42:45.277993 20956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:45.278079 20956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:45.546442 20956 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:35559
I20260619 06:42:45.546530 21092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:35559 every 8 connection(s)
I20260619 06:42:45.548358 20956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:42:45.557184 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 20956
I20260619 06:42:45.574239 20952 heartbeater.cc:503] Master 127.7.146.254:38963 was elected leader, sending a full tablet report...
Jun 19 06:42:45 dist-test-slave-qfk4 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851365, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:45.600394 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.552534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:36975 (local address 127.7.146.254:38963)
0619 06:42:45.578303 (+ 25769us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:45.578309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:45.578332 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:42:45.581288 (+ 2956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:45.581294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:45.581420 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:45.581502 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:45.583941 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.585550 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.588078 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.588722 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.588811 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.593224 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:45.593231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:45.593235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:45.593273 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:45.596531 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:45.597569 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:45.597574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:45.597578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:45.597709 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:45.598299 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:45.598303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:45.598306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:45.598723 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:45.598837 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:45.599795 (+ 958us) server_negotiation.cc:299] Negotiation successful
0619 06:42:45.600005 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25638,"thread_start_us":33666,"threads_started":1}
I20260619 06:42:45.602665 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.551862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38963 (local address 127.7.146.195:36975)
0619 06:42:45.575692 (+ 23830us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:45.575767 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:45.580751 (+ 4984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:45.582149 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:45.582162 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:45.582858 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:45.583707 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.583734 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.585785 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.585790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.587885 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.587899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.589185 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.589191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.589493 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.590286 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:45.590315 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:45.592952 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:45.596709 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:45.596719 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:45.596738 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:45.597391 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:45.597840 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:45.597844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:45.597847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:45.598109 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:45.598942 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:45.598949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:45.599395 (+ 446us) client_negotiation.cc:769] Sending connection context
0619 06:42:45.600633 (+ 1238us) client_negotiation.cc:240] Negotiation successful
0619 06:42:45.601210 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23469,"thread_start_us":25008,"threads_started":1}
I20260619 06:42:45.603060 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.558441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.7.146.254:38963)
0619 06:42:45.583811 (+ 25370us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:45.583817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:45.583835 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:42:45.583888 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:45.583893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:45.583986 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:45.584096 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:45.586417 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.587492 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.590416 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.591063 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.591171 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.595472 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:45.595479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:45.595484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:45.595536 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:45.598954 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:45.599886 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:45.599891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:45.599894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:45.600050 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:45.600636 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:45.600641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:45.600644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:45.601006 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:45.601264 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:45.602418 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0619 06:42:45.602673 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25233,"thread_start_us":19892,"threads_started":1}
I20260619 06:42:45.605674 21093 heartbeater.cc:347] Connected to a master server at 127.7.146.254:38963
I20260619 06:42:45.606016 21093 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:45.606959 21093 heartbeater.cc:511] Master 127.7.146.254:38963 requested a full tablet report, sending...
I20260619 06:42:45.608665 20610 ts_manager.cc:194] Registered new tserver with Master: c4a5bea8ce6444e9874a6d50d863db66 (127.7.146.195:35559)
I20260619 06:42:45.609740 20610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:36975
I20260619 06:42:45.620916 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:42:45.647353 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.624608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.7.146.254:38963)
0619 06:42:45.624793 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:45.624799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:45.631534 (+ 6735us) server_negotiation.cc:407] Connection header received
0619 06:42:45.631784 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:45.631789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:45.631875 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:45.631978 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:45.633895 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.634953 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.636908 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.637528 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.637648 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.641041 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:45.641047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:45.641052 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:45.641096 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:45.644354 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:45.645291 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:45.645297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:45.645301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:45.645458 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:45.645864 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:45.645869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:45.645872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:45.646219 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:45.646451 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:45.646791 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 06:42:45.647012 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:42:45.653304 20610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:42:45.664271 20610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:42:45.711715 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.679503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:35559 (local address 127.0.0.1:47274)
0619 06:42:45.695501 (+ 15998us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:45.695542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:45.695996 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:45.700744 (+ 4748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:45.700750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:45.700776 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:42:45.701178 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.701188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.704694 (+ 3506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.704700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.706943 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.706959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.710702 (+ 3743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.710709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.710930 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.710993 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 06:42:45.711132 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 06:42:45.711335 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15812,"thread_start_us":15677,"threads_started":1}
I20260619 06:42:45.712553 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.680173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.7.146.195:35559)
0619 06:42:45.692414 (+ 12241us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:45.692424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:45.695651 (+ 3227us) server_negotiation.cc:407] Connection header received
0619 06:42:45.699903 (+ 4252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:45.699913 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:45.700221 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:45.700383 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:42:45.701391 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.702998 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.708452 (+ 5454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.710106 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.710216 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.710484 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:45.712012 (+ 1528us) server_negotiation.cc:299] Negotiation successful
0619 06:42:45.712204 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12031,"thread_start_us":12247,"threads_started":1}
I20260619 06:42:45.715020 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.679403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:32817 (local address 127.0.0.1:48580)
0619 06:42:45.695738 (+ 16335us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:45.695755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:45.696210 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:45.697285 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:45.697293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:45.697325 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:42:45.697727 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.697739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.703147 (+ 5408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.703153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.708245 (+ 5092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.708262 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.714282 (+ 6020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.714288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.714457 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.714488 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:42:45.714571 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 06:42:45.714706 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15426,"thread_start_us":15294,"threads_started":1}
I20260619 06:42:45.715823 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.680109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.7.146.193:32817)
0619 06:42:45.694508 (+ 14399us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:45.694518 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:45.695834 (+ 1316us) server_negotiation.cc:407] Connection header received
0619 06:42:45.696402 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:45.696412 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:45.696743 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:45.696914 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:42:45.697934 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.699658 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.712889 (+ 13231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.714049 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.714192 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.714428 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:45.715354 (+ 926us) server_negotiation.cc:299] Negotiation successful
0619 06:42:45.715527 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14203,"thread_start_us":14013,"threads_started":1}
I20260619 06:42:45.718544 21028 tablet_service.cc:1467] Processing CreateTablet for tablet f1d0c87095a7451e96b58d75891fee7e (DEFAULT_TABLE table=test-table [id=21a8df19981f40a0aeea1fe455f4fa09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:42:45.719225 20742 tablet_service.cc:1467] Processing CreateTablet for tablet f1d0c87095a7451e96b58d75891fee7e (DEFAULT_TABLE table=test-table [id=21a8df19981f40a0aeea1fe455f4fa09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:42:45.720057 21028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d0c87095a7451e96b58d75891fee7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:42:45.720618 20742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d0c87095a7451e96b58d75891fee7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:42:45.725277 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.695657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:41063 (local address 127.0.0.1:36804)
0619 06:42:45.711262 (+ 15605us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:45.711279 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:45.711379 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:45.713788 (+ 2409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:45.713793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:45.713821 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:42:45.714098 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.714110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.715947 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.715952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.722162 (+ 6210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.722180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.724504 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.724511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.724707 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.724740 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:42:45.724823 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 06:42:45.724955 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15484,"thread_start_us":15335,"threads_started":1}
I20260619 06:42:45.726127 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.695995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.7.146.194:41063)
0619 06:42:45.713175 (+ 17180us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:45.713184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:45.713213 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:42:45.713317 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:45.713325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:45.713580 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:45.713778 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:42:45.714287 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.715783 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.722458 (+ 6675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.724011 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.724152 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.724362 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:45.725644 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0619 06:42:45.725804 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16997,"thread_start_us":16841,"threads_started":1}
I20260619 06:42:45.729918 20887 tablet_service.cc:1467] Processing CreateTablet for tablet f1d0c87095a7451e96b58d75891fee7e (DEFAULT_TABLE table=test-table [id=21a8df19981f40a0aeea1fe455f4fa09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:42:45.732709 20887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d0c87095a7451e96b58d75891fee7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:42:45.742370 21114 tablet_bootstrap.cc:492] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed: Bootstrap starting.
I20260619 06:42:45.742509 21115 tablet_bootstrap.cc:492] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: Bootstrap starting.
I20260619 06:42:45.747872 21114 tablet_bootstrap.cc:654] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:45.747872 21115 tablet_bootstrap.cc:654] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:45.749181 21115 log.cc:824] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:45.749183 21114 log.cc:824] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:45.758678 21114 tablet_bootstrap.cc:492] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed: No bootstrap required, opened a new log
I20260619 06:42:45.758736 21115 tablet_bootstrap.cc:492] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: No bootstrap required, opened a new log
I20260619 06:42:45.759074 21115 ts_tablet_manager.cc:1397] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20260619 06:42:45.759064 21114 ts_tablet_manager.cc:1397] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260619 06:42:45.761287 21116 tablet_bootstrap.cc:492] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7: Bootstrap starting.
I20260619 06:42:45.764438 21116 tablet_bootstrap.cc:654] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:45.765808 21116 log.cc:824] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:45.769598 21115 raft_consensus.cc:348] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451aa0dfc8a84be185b5e918704229ed" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 32817 } } peers { permanent_uuid: "c4a5bea8ce6444e9874a6d50d863db66" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35559 } } peers { permanent_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 41063 } }
I20260619 06:42:45.769801 21114 raft_consensus.cc:348] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451aa0dfc8a84be185b5e918704229ed" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 32817 } } peers { permanent_uuid: "c4a5bea8ce6444e9874a6d50d863db66" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35559 } } peers { permanent_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 41063 } }
I20260619 06:42:45.770083 21115 raft_consensus.cc:374] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:45.770175 21115 raft_consensus.cc:729] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a5bea8ce6444e9874a6d50d863db66, State: Initialized, Role: FOLLOWER
I20260619 06:42:45.770274 21114 raft_consensus.cc:374] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:45.770412 21114 raft_consensus.cc:729] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451aa0dfc8a84be185b5e918704229ed, State: Initialized, Role: FOLLOWER
I20260619 06:42:45.771148 21114 consensus_queue.cc:260] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451aa0dfc8a84be185b5e918704229ed" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 32817 } } peers { permanent_uuid: "c4a5bea8ce6444e9874a6d50d863db66" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35559 } } peers { permanent_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 41063 } }
I20260619 06:42:45.771539 21115 consensus_queue.cc:260] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451aa0dfc8a84be185b5e918704229ed" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 32817 } } peers { permanent_uuid: "c4a5bea8ce6444e9874a6d50d863db66" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35559 } } peers { permanent_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 41063 } }
I20260619 06:42:45.773363 21116 tablet_bootstrap.cc:492] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7: No bootstrap required, opened a new log
I20260619 06:42:45.773742 21116 ts_tablet_manager.cc:1397] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260619 06:42:45.779378 21114 ts_tablet_manager.cc:1428] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed: Time spent starting tablet: real 0.020s user 0.010s sys 0.008s
I20260619 06:42:45.780318 20807 heartbeater.cc:503] Master 127.7.146.254:38963 was elected leader, sending a full tablet report...
I20260619 06:42:45.781010 21115 ts_tablet_manager.cc:1428] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20260619 06:42:45.781649 21093 heartbeater.cc:503] Master 127.7.146.254:38963 was elected leader, sending a full tablet report...
I20260619 06:42:45.785115 21116 raft_consensus.cc:348] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451aa0dfc8a84be185b5e918704229ed" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 32817 } } peers { permanent_uuid: "c4a5bea8ce6444e9874a6d50d863db66" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35559 } } peers { permanent_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 41063 } }
I20260619 06:42:45.785657 21116 raft_consensus.cc:374] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:45.785771 21116 raft_consensus.cc:729] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f88f0ea0e040cd814178ef0bcdffa7, State: Initialized, Role: FOLLOWER
I20260619 06:42:45.786458 21116 consensus_queue.cc:260] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451aa0dfc8a84be185b5e918704229ed" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 32817 } } peers { permanent_uuid: "c4a5bea8ce6444e9874a6d50d863db66" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35559 } } peers { permanent_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 41063 } }
I20260619 06:42:45.788206 21116 ts_tablet_manager.cc:1428] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260619 06:42:45.840745 21094 tablet.cc:2369] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:42:45.886603 20809 tablet.cc:2369] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:42:45.942113 21121 raft_consensus.cc:482] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:42:45.942443 21121 raft_consensus.cc:504] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451aa0dfc8a84be185b5e918704229ed" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 32817 } } peers { permanent_uuid: "c4a5bea8ce6444e9874a6d50d863db66" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35559 } } peers { permanent_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 41063 } }
I20260619 06:42:45.944221 21121 leader_election.cc:288] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451aa0dfc8a84be185b5e918704229ed (127.7.146.193:32817), 02f88f0ea0e040cd814178ef0bcdffa7 (127.7.146.194:41063)
I20260619 06:42:45.949994 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.944721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:46521 (local address 127.7.146.193:32817)
0619 06:42:45.944922 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:45.944926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:45.944950 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:42:45.945147 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:45.945152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:45.945216 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:45.945308 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:42:45.945779 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.946539 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.948248 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.949116 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.949275 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.949361 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:45.949608 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 06:42:45.949735 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:42:45.950023 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.944649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:32817 (local address 127.7.146.195:46521)
0619 06:42:45.944824 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:45.944844 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:45.944968 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:45.945345 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:45.945350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:45.945379 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:42:45.945630 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.945641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.946682 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.946688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.948068 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.948085 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.949277 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.949283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.949457 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.949482 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:42:45.949575 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:42:45.949709 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":21}
I20260619 06:42:45.950614 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.945322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:46757 (local address 127.7.146.194:41063)
0619 06:42:45.945486 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:45.945492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:45.945931 (+ 439us) server_negotiation.cc:407] Connection header received
0619 06:42:45.946082 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:45.946087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:45.946150 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:45.946222 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:42:45.946713 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.947413 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.949054 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.949791 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.949931 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.949999 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:45.950304 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 06:42:45.950433 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 06:42:45.950733 21123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.945063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:41063 (local address 127.7.146.195:46757)
0619 06:42:45.945831 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:45.945844 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:45.945951 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:45.946259 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:45.946263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:45.946284 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:42:45.946577 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.946586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.947557 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.947561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.948906 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:45.948918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.949971 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.949977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:45.950145 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.950181 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:42:45.950267 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 06:42:45.950378 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":3,"thread_start_us":483,"threads_started":1}
I20260619 06:42:45.951248 20762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d0c87095a7451e96b58d75891fee7e" candidate_uuid: "c4a5bea8ce6444e9874a6d50d863db66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451aa0dfc8a84be185b5e918704229ed" is_pre_election: true
I20260619 06:42:45.951757 20762 raft_consensus.cc:2454] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4a5bea8ce6444e9874a6d50d863db66 in term 0.
I20260619 06:42:45.951670 20907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d0c87095a7451e96b58d75891fee7e" candidate_uuid: "c4a5bea8ce6444e9874a6d50d863db66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" is_pre_election: true
I20260619 06:42:45.952126 20907 raft_consensus.cc:2454] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4a5bea8ce6444e9874a6d50d863db66 in term 0.
I20260619 06:42:45.952498 20974 leader_election.cc:302] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451aa0dfc8a84be185b5e918704229ed, c4a5bea8ce6444e9874a6d50d863db66; no voters:
I20260619 06:42:45.952935 21121 raft_consensus.cc:2781] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:42:45.953118 21121 raft_consensus.cc:482] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:42:45.953239 21121 raft_consensus.cc:3037] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:45.955466 21121 raft_consensus.cc:504] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451aa0dfc8a84be185b5e918704229ed" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 32817 } } peers { permanent_uuid: "c4a5bea8ce6444e9874a6d50d863db66" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35559 } } peers { permanent_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 41063 } }
I20260619 06:42:45.956065 21121 leader_election.cc:288] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [CANDIDATE]: Term 1 election: Requested vote from peers 451aa0dfc8a84be185b5e918704229ed (127.7.146.193:32817), 02f88f0ea0e040cd814178ef0bcdffa7 (127.7.146.194:41063)
I20260619 06:42:45.956557 20762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d0c87095a7451e96b58d75891fee7e" candidate_uuid: "c4a5bea8ce6444e9874a6d50d863db66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451aa0dfc8a84be185b5e918704229ed"
I20260619 06:42:45.956636 20907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d0c87095a7451e96b58d75891fee7e" candidate_uuid: "c4a5bea8ce6444e9874a6d50d863db66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f88f0ea0e040cd814178ef0bcdffa7"
I20260619 06:42:45.956805 20762 raft_consensus.cc:3037] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:45.956812 20907 raft_consensus.cc:3037] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:45.958953 20907 raft_consensus.cc:2454] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4a5bea8ce6444e9874a6d50d863db66 in term 1.
I20260619 06:42:45.958981 20762 raft_consensus.cc:2454] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4a5bea8ce6444e9874a6d50d863db66 in term 1.
I20260619 06:42:45.959362 20974 leader_election.cc:302] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451aa0dfc8a84be185b5e918704229ed, c4a5bea8ce6444e9874a6d50d863db66; no voters:
I20260619 06:42:45.959669 21121 raft_consensus.cc:2781] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:42:45.960093 21121 raft_consensus.cc:686] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 1 LEADER]: Becoming Leader. State: Replica: c4a5bea8ce6444e9874a6d50d863db66, State: Running, Role: LEADER
I20260619 06:42:45.960528 21121 consensus_queue.cc:237] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451aa0dfc8a84be185b5e918704229ed" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 32817 } } peers { permanent_uuid: "c4a5bea8ce6444e9874a6d50d863db66" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35559 } } peers { permanent_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 41063 } }
I20260619 06:42:45.965081 20610 catalog_manager.cc:5495] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4a5bea8ce6444e9874a6d50d863db66 (127.7.146.195). New cstate: current_term: 1 leader_uuid: "c4a5bea8ce6444e9874a6d50d863db66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "451aa0dfc8a84be185b5e918704229ed" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 32817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4a5bea8ce6444e9874a6d50d863db66" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 35559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 41063 } health_report { overall_health: UNKNOWN } } }
I20260619 06:42:45.987448 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:45.982257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47288 (local address 127.7.146.195:35559)
0619 06:42:45.982452 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:45.982456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:45.982568 (+ 112us) server_negotiation.cc:407] Connection header received
0619 06:42:45.982751 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:45.982755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:45.982841 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:45.982974 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:42:45.983386 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.984110 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.985558 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:45.985949 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:45.986042 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:45.986443 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:42:45.986948 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:42:45.987056 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:45.987192 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 06:42:45.987255 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:42:45.996163 20762 raft_consensus.cc:1261] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed [term 1 FOLLOWER]: Refusing update from remote peer c4a5bea8ce6444e9874a6d50d863db66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:42:45.996148 20907 raft_consensus.cc:1261] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7 [term 1 FOLLOWER]: Refusing update from remote peer c4a5bea8ce6444e9874a6d50d863db66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:42:45.997133 21124 consensus_queue.cc:1035] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f88f0ea0e040cd814178ef0bcdffa7" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 41063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:42:45.997473 21121 consensus_queue.cc:1035] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "451aa0dfc8a84be185b5e918704229ed" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 32817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:42:46.008319 21127 mvcc.cc:205] Tried to move back new op lower bound from 7298463195109986304 to 7298463194977955840. Current Snapshot: MvccSnapshot[applied={T|T < 7298463195109986304}]
I20260619 06:42:46.026353 20609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:42:46.026690 20609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:42:46.030318 20609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec36bb821331475bba28aaec20f3a153: Sending DeleteTablet for 3 replicas of tablet f1d0c87095a7451e96b58d75891fee7e
I20260619 06:42:46.031545 20742 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d0c87095a7451e96b58d75891fee7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:42:46 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:48580
I20260619 06:42:46.031540 21028 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d0c87095a7451e96b58d75891fee7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:42:46 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:47274
I20260619 06:42:46.032483 20887 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d0c87095a7451e96b58d75891fee7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:42:46 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:36804
I20260619 06:42:46.037065 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 20647
I20260619 06:42:46.041641 21139 tablet_replica.cc:330] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7: stopping tablet replica
I20260619 06:42:46.042179 21139 raft_consensus.cc:2229] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:42:46.042752 21139 raft_consensus.cc:2258] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:42:46.048278 21137 tablet_replica.cc:330] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: stopping tablet replica
I20260619 06:42:46.048718 21137 raft_consensus.cc:2229] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:42:46.050500 21139 ts_tablet_manager.cc:1905] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:42:46.050503 21137 raft_consensus.cc:2258] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:42:46.051753 21138 tablet_replica.cc:330] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed: stopping tablet replica
I20260619 06:42:46.052168 21138 raft_consensus.cc:2229] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:42:46.052680 21138 raft_consensus.cc:2258] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:42:46.054572 21137 ts_tablet_manager.cc:1905] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:42:46.054807 21138 ts_tablet_manager.cc:1905] T f1d0c87095a7451e96b58d75891fee7e P 451aa0dfc8a84be185b5e918704229ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:42:46.057650 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:46.038330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.7.146.193:32817)
0619 06:42:46.038506 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:46.038512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:46.038531 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:42:46.038595 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:46.038600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:46.038678 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:46.038780 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:46.040872 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:46.042020 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:46.044405 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:46.045109 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:46.045230 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:46.048811 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:46.048829 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:46.048848 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:46.048892 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:46.052881 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:46.054247 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:46.054256 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:46.054260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:46.054424 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:46.054957 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:46.054962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:46.054965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:46.055404 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:46.055740 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:46.057025 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0619 06:42:46.057257 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:42:46.058864 20802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:46.061853 21137 ts_tablet_manager.cc:1918] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:42:46.062103 21137 log.cc:1196] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/f1d0c87095a7451e96b58d75891fee7e
I20260619 06:42:46.062713 21137 ts_tablet_manager.cc:1939] T f1d0c87095a7451e96b58d75891fee7e P c4a5bea8ce6444e9874a6d50d863db66: Deleting consensus metadata
I20260619 06:42:46.064074 21139 ts_tablet_manager.cc:1918] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:42:46.064285 21139 log.cc:1196] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/f1d0c87095a7451e96b58d75891fee7e
I20260619 06:42:46.064810 21139 ts_tablet_manager.cc:1939] T f1d0c87095a7451e96b58d75891fee7e P 02f88f0ea0e040cd814178ef0bcdffa7: Deleting consensus metadata
I20260619 06:42:46.064751 20596 catalog_manager.cc:4841] TS c4a5bea8ce6444e9874a6d50d863db66 (127.7.146.195:35559): tablet f1d0c87095a7451e96b58d75891fee7e (table test-table [id=21a8df19981f40a0aeea1fe455f4fa09]) successfully deleted
I20260619 06:42:46.066570 20595 catalog_manager.cc:4841] TS 02f88f0ea0e040cd814178ef0bcdffa7 (127.7.146.194:41063): tablet f1d0c87095a7451e96b58d75891fee7e (table test-table [id=21a8df19981f40a0aeea1fe455f4fa09]) successfully deleted
I20260619 06:42:46.155062 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 20647
W20260619 06:42:46.176707 20595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.146.193:32817 (error 108)
I20260619 06:42:46.177995 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 20808
I20260619 06:42:46.178504 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:46.178017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:32817 (local address 127.0.0.1:48592)
0619 06:42:46.178207 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:46.178335 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 06:42:46.179011 20595 catalog_manager.cc:4584] TS 451aa0dfc8a84be185b5e918704229ed (127.7.146.193:32817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1d0c87095a7451e96b58d75891fee7e: Network error: Client connection negotiation failed: client connection to 127.7.146.193:32817: connect: Connection refused (error 111)
I20260619 06:42:46.192886 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:46.178730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.7.146.194:41063)
0619 06:42:46.178876 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:46.178880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:46.179055 (+ 175us) server_negotiation.cc:407] Connection header received
0619 06:42:46.179216 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:46.179221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:46.179308 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:46.179393 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:46.181108 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:46.181828 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:46.183745 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:46.184174 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:46.184269 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:46.187375 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:46.187391 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:46.187406 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:46.187444 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:46.190158 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:46.190950 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:46.190956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:46.190959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:46.191081 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:46.191453 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:46.191455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:46.191457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:46.191802 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:46.192045 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:46.192384 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 06:42:46.192542 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:42:46.193572 20947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:46.232915 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:46.232308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:32817 (local address 127.0.0.1:48602)
0619 06:42:46.232622 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:46.232746 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260619 06:42:46.234081 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:46.233660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:32817 (local address 127.0.0.1:48610)
0619 06:42:46.233815 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:46.233909 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:42:46.283074 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 20808
I20260619 06:42:46.294850 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:46.294198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:32817 (local address 127.0.0.1:48622)
0619 06:42:46.294506 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:46.294648 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260619 06:42:46.296060 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:46.295672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:32817 (local address 127.0.0.1:48628)
0619 06:42:46.295832 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:46.295921 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 06:42:46.296852 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 20956
I20260619 06:42:46.313511 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:46.297600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.7.146.195:35559)
0619 06:42:46.297771 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:46.297775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:46.297924 (+ 149us) server_negotiation.cc:407] Connection header received
0619 06:42:46.298158 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:46.298162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:46.298227 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:46.298301 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:46.300011 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:46.300875 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:46.302724 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:46.303176 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:46.303265 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:46.306728 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:46.306740 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:46.306758 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:46.306796 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:46.310441 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:46.311204 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:46.311213 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:46.311217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:46.311408 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:46.311775 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:46.311781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:46.311784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:46.312275 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:46.312728 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:46.312886 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 06:42:46.313117 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:42:46.314297 21088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:46.377390 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:46.376839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:32817 (local address 127.0.0.1:48642)
0619 06:42:46.377098 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:46.377207 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 06:42:46.378446 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:46.378064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:32817 (local address 127.0.0.1:48652)
0619 06:42:46.378258 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:46.378332 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 06:42:46.432809 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 20956
I20260619 06:42:46.448371 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 20577
I20260619 06:42:46.449549 20640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:46.560259 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:46.559534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:32817 (local address 127.0.0.1:48656)
0619 06:42:46.559710 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:46.559820 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 06:42:46.560504 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 20577
I20260619 06:42:46.577598 7755 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 06:42:46.912689 7755 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 06:42:46.912887 7755 mini_ranger.cc:66] Stopping Ranger...
I20260619 06:42:47.248013 7755 mini_ranger.cc:68] Stopped Ranger
2026-06-19 06:42:47.248 UTC [19630] LOG: received smart shutdown request
2026-06-19 06:42:47.251 UTC [19630] LOG: background worker "logical replication launcher" (PID 19639) exited with exit code 1
2026-06-19 06:42:47.251 UTC [19634] LOG: shutting down
2026-06-19 06:42:47.403 UTC [19630] LOG: database system is shut down
2026-06-19T06:42:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51855 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:42:47 dist-test-slave-qfk4 krb5kdc[21153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:42:47 dist-test-slave-qfk4 krb5kdc[21153](info): set up 2 sockets
Jun 19 06:42:47 dist-test-slave-qfk4 krb5kdc[21153](info): commencing operation
krb5kdc: starting...
W20260619 06:42:48.516762 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:42:48 dist-test-slave-qfk4 krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:42:48.727527 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34335
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:34335,127.7.146.253:34251,127.7.146.252:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:42:49.041170 21166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:49.041467 21166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:49.041594 21166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:49.048923 21166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:42:49.049031 21166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:49.049088 21166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:49.049170 21166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:42:49.049220 21166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:42:49.059070 21166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:34335,127.7.146.253:34251,127.7.146.252:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:49.060379 21166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:49.061668 21166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:49.073499 21172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:49.073586 21171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:49.074224 21174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:49.075184 21166 server_base.cc:1034] running on GCE node
I20260619 06:42:49.076089 21166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:49.077574 21166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:49.078868 21166 hybrid_clock.cc:648] HybridClock initialized: now 1781851369078812 us; error 39 us; skew 500 ppm
I20260619 06:42:49.079339 21166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:49 dist-test-slave-qfk4 krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851369, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:49.083159 21166 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:42:49.086048 21166 webserver.cc:460] Webserver started at http://127.7.146.254:43023/ using document root <none> and password file <none>
I20260619 06:42:49.086982 21166 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:49.087113 21166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:49.087414 21166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:42:49.091631 21166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "e29ed3a9aa7f4132a7d8b5e610887147"
format_stamp: "Formatted at 2026-06-19 06:42:49 on dist-test-slave-qfk4"
server_key: "992d9d9c5b9fc511e8bd2b5872831c7e"
server_key_iv: "c1e37ed5e6e81faa5870cac03bc90f61"
server_key_version: "encryptionkey@0"
I20260619 06:42:49.092737 21166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "e29ed3a9aa7f4132a7d8b5e610887147"
format_stamp: "Formatted at 2026-06-19 06:42:49 on dist-test-slave-qfk4"
server_key: "992d9d9c5b9fc511e8bd2b5872831c7e"
server_key_iv: "c1e37ed5e6e81faa5870cac03bc90f61"
server_key_version: "encryptionkey@0"
I20260619 06:42:49.099123 21166 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 06:42:49.103108 21181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:49.104840 21166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:42:49.105006 21166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "e29ed3a9aa7f4132a7d8b5e610887147"
format_stamp: "Formatted at 2026-06-19 06:42:49 on dist-test-slave-qfk4"
server_key: "992d9d9c5b9fc511e8bd2b5872831c7e"
server_key_iv: "c1e37ed5e6e81faa5870cac03bc90f61"
server_key_version: "encryptionkey@0"
I20260619 06:42:49.105208 21166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:49.122406 21166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:49.126435 21166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:49.126703 21166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:49.143952 21166 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:34335
I20260619 06:42:49.143986 21232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:34335 every 8 connection(s)
I20260619 06:42:49.145351 21166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:42:49.147401 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 21166
I20260619 06:42:49.147630 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:42:49.148404 7755 external_mini_cluster.cc:1428] Setting key b307b7b671b5ef3bc297017258a93654
I20260619 06:42:49.149693 21233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:42:49.157984 21233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.146.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.253@KRBTEST.COM" created.
Jun 19 06:42:49 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851369, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:49.181442 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.161275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.7.146.254:34335)
0619 06:42:49.163272 (+ 1997us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:49.163278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:49.163321 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:42:49.165357 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:49.165363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:49.165606 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:49.165768 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:49.167355 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.168552 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.169938 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.170522 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.170683 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:49.174328 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:49.174337 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:49.174350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:49.174382 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:49.177248 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:49.178095 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:49.178102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:49.178104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:49.178252 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:49.178656 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:49.178660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:49.178663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:49.179344 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:49.179537 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:49.180033 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 06:42:49.180309 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1812,"spinlock_wait_cycles":75136,"thread_start_us":1570,"threads_started":1}
I20260619 06:42:49.181468 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.161275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34335 (local address 127.0.0.1:58678)
0619 06:42:49.162247 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.162297 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:49.164932 (+ 2635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:49.165852 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:49.165860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:49.166644 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:49.167153 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:49.167181 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.168710 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.168715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:49.169811 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:49.169823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.170751 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.170758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:49.170909 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:49.171524 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:49.171546 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:49.174055 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:49.177392 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:49.177405 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:49.177418 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:49.177950 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:49.178376 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:49.178381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:49.178385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:49.178543 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:49.179479 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:49.179486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:49.179705 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 06:42:49.179924 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 06:42:49.180503 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":132,"thread_start_us":413,"threads_started":1}
I20260619 06:42:49.184481 21233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:42:49.185485 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.185117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:56986)
0619 06:42:49.185265 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.185381 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 06:42:49.185719 21183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
I20260619 06:42:49.187152 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.186824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57000)
0619 06:42:49.186951 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.187019 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:42:49.187351 21233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:34251: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab.
Entry for principal kudu/127.7.146.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab.
I20260619 06:42:49.199664 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.253:34251
--webserver_interface=127.7.146.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab
--principal=kudu/127.7.146.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:34335,127.7.146.253:34251,127.7.146.252:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:42:49.236693 21233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } attempt: 1
I20260619 06:42:49.237836 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.237410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57014)
0619 06:42:49.237597 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.237711 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 06:42:49.238831 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.238481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57022)
0619 06:42:49.238607 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.238679 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:42:49.239017 21233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:34251: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
I20260619 06:42:49.307258 21233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } attempt: 2
I20260619 06:42:49.308339 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.307910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57024)
0619 06:42:49.308101 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.308198 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:42:49.309448 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.309098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57038)
0619 06:42:49.309218 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.309306 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 06:42:49.309639 21233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:34251: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
I20260619 06:42:49.400856 21233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } attempt: 3
I20260619 06:42:49.402055 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.401630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57042)
0619 06:42:49.401819 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.401925 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 06:42:49.403107 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.402784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57044)
0619 06:42:49.402899 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.402970 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 06:42:49.403318 21233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:34251: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
W20260619 06:42:49.521035 21240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:49.521319 21240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:49.521432 21240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:49.528625 21240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:42:49.528726 21240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:49.528786 21240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:49.528864 21240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:42:49.528921 21240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:42:49.538735 21240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:34335,127.7.146.253:34251,127.7.146.252:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab
--principal=kudu/127.7.146.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.253:34251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.146.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:49.540045 21240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:49.541407 21240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:42:49.546556 21233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } attempt: 4
I20260619 06:42:49.547731 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.547238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57052)
0619 06:42:49.547506 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.547599 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260619 06:42:49.548851 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.548432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57060)
0619 06:42:49.548601 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.548729 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:42:49.549057 21233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:34251: Network error: Client connection negotiation failed: client connection to 127.7.146.253:34251: connect: Connection refused (error 111)
W20260619 06:42:49.553104 21248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:49.553138 21246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:49.553795 21245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:49.554417 21240 server_base.cc:1034] running on GCE node
I20260619 06:42:49.555202 21240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:49.556664 21240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:49.557868 21240 hybrid_clock.cc:648] HybridClock initialized: now 1781851369557808 us; error 54 us; skew 500 ppm
I20260619 06:42:49.558197 21240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:49 dist-test-slave-qfk4 krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851369, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:49.564141 21240 init.cc:377] Logged in from keytab as kudu/127.7.146.253@KRBTEST.COM (short username kudu)
I20260619 06:42:49.565717 21240 webserver.cc:460] Webserver started at http://127.7.146.253:42863/ using document root <none> and password file <none>
I20260619 06:42:49.566368 21240 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:49.566469 21240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:49.566726 21240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:42:49.569636 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/data/instance:
uuid: "548eec022f844a1b994c707b437a8fda"
format_stamp: "Formatted at 2026-06-19 06:42:49 on dist-test-slave-qfk4"
server_key: "2a8ca18a055b9a3f849e73bd4d646c18"
server_key_iv: "54a9bd25466bdceb199cd603c75d6772"
server_key_version: "encryptionkey@0"
I20260619 06:42:49.570359 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/wal/instance:
uuid: "548eec022f844a1b994c707b437a8fda"
format_stamp: "Formatted at 2026-06-19 06:42:49 on dist-test-slave-qfk4"
server_key: "2a8ca18a055b9a3f849e73bd4d646c18"
server_key_iv: "54a9bd25466bdceb199cd603c75d6772"
server_key_version: "encryptionkey@0"
I20260619 06:42:49.575276 21240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:42:49.578579 21255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:49.579937 21240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:42:49.580102 21240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/wal
uuid: "548eec022f844a1b994c707b437a8fda"
format_stamp: "Formatted at 2026-06-19 06:42:49 on dist-test-slave-qfk4"
server_key: "2a8ca18a055b9a3f849e73bd4d646c18"
server_key_iv: "54a9bd25466bdceb199cd603c75d6772"
server_key_version: "encryptionkey@0"
I20260619 06:42:49.580292 21240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:49.600174 21240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:49.604616 21240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:49.604890 21240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:49.622339 21240 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.253:34251
I20260619 06:42:49.622350 21306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.253:34251 every 8 connection(s)
I20260619 06:42:49.623648 21240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
I20260619 06:42:49.627219 21307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:42:49.630321 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 21240
I20260619 06:42:49.630522 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-1/wal/instance
I20260619 06:42:49.631050 7755 external_mini_cluster.cc:1428] Setting key 00a68ba02f71b015aeb45997674e4632
I20260619 06:42:49.635319 21307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:42:49 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851369, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.252@KRBTEST.COM" created.
I20260619 06:42:49.657104 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.638553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.7.146.254:34335)
0619 06:42:49.638775 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:49.638781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:49.639979 (+ 1198us) server_negotiation.cc:407] Connection header received
0619 06:42:49.642141 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:49.642147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:49.642234 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:49.642337 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:49.643903 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.644827 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.646363 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.646954 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.647093 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:49.651020 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:49.651025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:49.651028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:49.651063 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:49.653971 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:49.654849 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:49.654853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:49.654856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:49.654975 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:49.655392 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:49.655395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:49.655397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:49.655693 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:49.655803 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:49.656697 (+ 894us) server_negotiation.cc:299] Negotiation successful
0619 06:42:49.656871 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:42:49.658500 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.639030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34335 (local address 127.0.0.1:58690)
0619 06:42:49.639812 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.639863 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:49.641801 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:49.642598 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:49.642609 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:49.643162 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:49.643727 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:49.643745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.644979 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.644983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:49.646231 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:49.646243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.647223 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.647232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:49.647499 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:49.648225 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:49.648248 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:49.650743 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:49.654113 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:49.654123 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:49.654141 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:49.654731 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:49.655069 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:49.655073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:49.655076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:49.655302 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:49.655798 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:49.655805 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:49.656249 (+ 444us) client_negotiation.cc:769] Sending connection context
0619 06:42:49.656552 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 06:42:49.657101 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":328,"threads_started":1}
I20260619 06:42:49.660284 21307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:42:49 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851369, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
I20260619 06:42:49.676280 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.661189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57076)
0619 06:42:49.661367 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.661387 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:49.661497 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:49.662445 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:49.662448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:49.662949 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:49.663104 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:49.663113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.664958 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.664962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:49.665825 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:49.665836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.667001 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.667006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:49.667146 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:49.667724 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:49.667742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:49.670049 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:49.674006 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:49.674010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:49.674012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:49.674347 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:49.674697 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:49.674701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:49.674703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:49.674872 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:49.675493 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:49.675496 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:49.675619 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 06:42:49.675760 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:42:49.675969 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 06:42:49.676399 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.661257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.7.146.253:34251)
0619 06:42:49.661946 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:49.661953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:49.661982 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:42:49.662032 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:49.662039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:49.662294 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:49.662434 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:49.663265 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.664805 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.665960 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.666853 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.666966 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:49.670259 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:49.670274 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:49.670293 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:49.670335 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:49.673867 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:49.674461 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:49.674467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:49.674470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:49.674588 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:49.674958 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:49.674961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:49.674963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:49.675340 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:49.675523 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:49.675879 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 06:42:49.676063 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":382,"threads_started":1}
I20260619 06:42:49.678752 21307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:42:49.679826 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.679382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50186)
0619 06:42:49.679559 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.679673 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 06:42:49.680110 21257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
I20260619 06:42:49.681737 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.681399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50192)
0619 06:42:49.681521 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.681609 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 06:42:49.681921 21307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:38411: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab.
Entry for principal kudu/127.7.146.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab.
I20260619 06:42:49.684366 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.252:38411
--webserver_interface=127.7.146.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab
--principal=kudu/127.7.146.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:34335,127.7.146.253:34251,127.7.146.252:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:42:49.731207 21307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } attempt: 1
I20260619 06:42:49.732445 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.731846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50206)
0619 06:42:49.732086 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.732251 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 06:42:49.733613 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.733215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50222)
0619 06:42:49.733391 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.733479 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 06:42:49.733853 21307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:38411: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
I20260619 06:42:49.802062 21307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } attempt: 2
I20260619 06:42:49.803251 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.802691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50224)
0619 06:42:49.802919 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.803035 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 06:42:49.804260 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.803872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50240)
0619 06:42:49.804023 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.804108 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:42:49.804481 21307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:38411: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
I20260619 06:42:49.848330 21233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } attempt: 5
Jun 19 06:42:49 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851369, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
I20260619 06:42:49.862495 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.849331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.7.146.253:34251)
0619 06:42:49.849548 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:49.849555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:49.849615 (+ 60us) server_negotiation.cc:407] Connection header received
0619 06:42:49.849688 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:49.849694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:49.849807 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:49.849926 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:49.850729 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.851815 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.853122 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.853799 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.853931 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:49.857208 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:49.857213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:49.857216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:49.857251 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:49.859773 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:49.860694 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:49.860698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:49.860701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:49.860824 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:49.861239 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:49.861242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:49.861244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:49.861527 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:49.861661 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:49.862087 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 06:42:49.862264 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:42:49.862618 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.849167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57092)
0619 06:42:49.849393 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.849415 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:49.849531 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:49.849989 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:49.849995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:49.850362 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:49.850537 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:49.850545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.851998 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.852006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:49.852953 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:49.852973 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:49.853942 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:49.853947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:49.854099 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:49.854701 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:49.854719 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:49.857029 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:49.859938 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:49.859944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:49.859949 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:49.860543 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:49.860933 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:49.860938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:49.860942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:49.861136 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:49.861639 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:49.861645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:49.861844 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 06:42:49.861990 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 06:42:49.862297 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 06:42:49.863768 21233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:42:49.864899 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.864403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50246)
0619 06:42:49.864648 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.864734 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260619 06:42:49.865816 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.865498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50248)
0619 06:42:49.865637 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.865720 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:42:49.866004 21233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:38411: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
I20260619 06:42:49.895756 21307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } attempt: 3
I20260619 06:42:49.897042 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.896502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50254)
0619 06:42:49.896744 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.896855 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 06:42:49.898202 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.897696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50266)
0619 06:42:49.897927 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.898025 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260619 06:42:49.898445 21307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:38411: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
I20260619 06:42:49.917210 21233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } attempt: 1
I20260619 06:42:49.918287 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.917836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50274)
0619 06:42:49.918023 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.918134 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:42:49.919260 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.918966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50286)
0619 06:42:49.919080 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.919161 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 06:42:49.919458 21233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:38411: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
I20260619 06:42:49.987665 21233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } attempt: 2
I20260619 06:42:49.988822 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.988329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50300)
0619 06:42:49.988530 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.988662 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 06:42:49.989729 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:49.989436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50314)
0619 06:42:49.989557 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:49.989629 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 06:42:49.989924 21233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:38411: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
W20260619 06:42:50.004068 21314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:50.004348 21314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:50.004469 21314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:50.012243 21314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:42:50.012377 21314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:50.012430 21314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:50.012504 21314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:42:50.012555 21314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:42:50.023095 21314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:34335,127.7.146.253:34251,127.7.146.252:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab
--principal=kudu/127.7.146.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.252:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.146.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:50.024410 21314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:50.025712 21314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:50.037222 21319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:50.037195 21322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:50.037276 21320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:50.038035 21314 server_base.cc:1034] running on GCE node
I20260619 06:42:50.038815 21314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:50.040235 21314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:50.041424 21314 hybrid_clock.cc:648] HybridClock initialized: now 1781851370041378 us; error 45 us; skew 500 ppm
I20260619 06:42:50.041653 21307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } attempt: 4
I20260619 06:42:50.041836 21314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:42:50.042783 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.042261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50318)
0619 06:42:50.042497 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:50.042597 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:42:50.043795 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.043423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50320)
0619 06:42:50.043576 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:50.043662 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:42:50.044047 21307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:38411: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Jun 19 06:42:50 dist-test-slave-qfk4 krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851370, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:50.045452 21314 init.cc:377] Logged in from keytab as kudu/127.7.146.252@KRBTEST.COM (short username kudu)
I20260619 06:42:50.046950 21314 webserver.cc:460] Webserver started at http://127.7.146.252:33197/ using document root <none> and password file <none>
I20260619 06:42:50.047609 21314 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:50.047722 21314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:50.048007 21314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:42:50.050987 21314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/data/instance:
uuid: "71b25abd9c4d41a08d4398f7bdf76794"
format_stamp: "Formatted at 2026-06-19 06:42:50 on dist-test-slave-qfk4"
server_key: "81fcd6aaa13179ef25b0dc0b1acc6fbb"
server_key_iv: "5b025a2f971783e5e8813117df8c5cf7"
server_key_version: "encryptionkey@0"
I20260619 06:42:50.051779 21314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/wal/instance:
uuid: "71b25abd9c4d41a08d4398f7bdf76794"
format_stamp: "Formatted at 2026-06-19 06:42:50 on dist-test-slave-qfk4"
server_key: "81fcd6aaa13179ef25b0dc0b1acc6fbb"
server_key_iv: "5b025a2f971783e5e8813117df8c5cf7"
server_key_version: "encryptionkey@0"
I20260619 06:42:50.057066 21314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:42:50.060523 21329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:50.062091 21314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:42:50.062271 21314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/wal
uuid: "71b25abd9c4d41a08d4398f7bdf76794"
format_stamp: "Formatted at 2026-06-19 06:42:50 on dist-test-slave-qfk4"
server_key: "81fcd6aaa13179ef25b0dc0b1acc6fbb"
server_key_iv: "5b025a2f971783e5e8813117df8c5cf7"
server_key_version: "encryptionkey@0"
I20260619 06:42:50.062502 21314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:50.083818 21314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:50.087736 21314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:50.087997 21314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:50.096151 21233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } attempt: 3
I20260619 06:42:50.097390 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.096859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50322)
0619 06:42:50.097122 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:50.097207 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 06:42:50.098608 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.098168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50334)
0619 06:42:50.098366 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:50.098468 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 06:42:50.098874 21233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:38411: Network error: Client connection negotiation failed: client connection to 127.7.146.252:38411: connect: Connection refused (error 111)
I20260619 06:42:50.106026 21314 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.252:38411
I20260619 06:42:50.106050 21380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.252:38411 every 8 connection(s)
I20260619 06:42:50.107391 21314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
I20260619 06:42:50.111213 21381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:42:50.114626 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 21314
I20260619 06:42:50.114825 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851174422545-7755-0/minicluster-data/master-2/wal/instance
I20260619 06:42:50.115382 7755 external_mini_cluster.cc:1428] Setting key abd6fc808b1b53c50f9af62130e64591
I20260619 06:42:50.119006 21381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:42:50 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
Jun 19 06:42:50 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851370, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:50.135465 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.122223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.7.146.254:34335)
0619 06:42:50.122389 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:50.122393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:50.123036 (+ 643us) server_negotiation.cc:407] Connection header received
0619 06:42:50.123213 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:50.123217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:50.123277 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:50.123355 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:50.124432 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.125181 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.126513 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.126926 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.127028 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.130334 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:50.130338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:50.130341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:50.130375 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:50.132704 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.133355 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.133361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.133365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.133501 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.133933 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.133938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.133941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.134206 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:50.134345 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:50.134874 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 06:42:50.135108 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:42:50.136541 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.117432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.7.146.252:38411)
0619 06:42:50.118165 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:50.118178 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:50.118217 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:42:50.119814 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:50.119837 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:50.120273 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:50.120651 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:50.122113 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.123816 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.125207 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.125892 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.126068 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.130071 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:50.130079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:50.130092 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:50.130127 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:50.132794 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.133484 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.133491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.133493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.133651 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.134012 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.134016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.134018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.134526 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:50.134813 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:50.135169 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 06:42:50.135413 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":332,"threads_started":1}
I20260619 06:42:50.136610 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.122196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34335 (local address 127.0.0.1:58700)
0619 06:42:50.122900 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:50.122941 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:50.123088 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:50.123440 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:50.123448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:50.124033 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:50.124291 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:50.124299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.125321 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.125325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:50.126385 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:50.126396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.127128 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.127135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:50.127274 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.127809 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:50.127830 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:50.130145 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:50.132823 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:50.132835 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:50.132847 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:50.133245 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:50.133603 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:50.133608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:50.133611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:50.133831 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:50.134323 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:50.134332 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:50.134536 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 06:42:50.134767 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 06:42:50.135268 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"spinlock_wait_cycles":29312,"thread_start_us":381,"threads_started":1}
I20260619 06:42:50.138207 21381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:42:50 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851370, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
Jun 19 06:42:50 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:50.153101 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.139012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.7.146.253:34251)
0619 06:42:50.139206 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:50.139212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:50.139424 (+ 212us) server_negotiation.cc:407] Connection header received
0619 06:42:50.139571 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:50.139577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:50.139660 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:50.139760 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:50.140621 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.141674 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.143303 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.143782 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.143892 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.147085 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:50.147089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:50.147093 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:50.147136 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:50.150238 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.151050 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.151055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.151059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.151216 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.151665 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.151671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.151674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.152017 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:50.152166 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:50.152640 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 06:42:50.152820 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:42:50.153278 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.139026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:34251 (local address 127.0.0.1:57100)
0619 06:42:50.139336 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:50.139355 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:50.139467 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:50.139785 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:50.139789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:50.140257 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:50.140441 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:50.140452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.141836 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.141840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:50.143185 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:50.143200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.143919 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.143924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:50.144131 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.144684 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:50.144701 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:50.146899 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:50.150387 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:50.150393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:50.150397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:50.150919 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:50.151338 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:50.151344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:50.151348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:50.151541 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:50.152136 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:50.152141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:50.152320 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 06:42:50.152472 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 06:42:50.152825 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260619 06:42:50.154393 21381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:42:50.157778 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.139960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.7.146.254:34335)
0619 06:42:50.140136 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:50.140141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:50.140170 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:42:50.140260 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:50.140265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:50.140331 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:50.140427 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:50.141959 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.143085 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.145020 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.145674 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.145800 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.150770 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:50.150775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:50.150779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:50.150811 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:50.154770 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.155663 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.155669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.155672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.155823 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.156321 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.156327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.156330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.156723 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:50.157033 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:50.157227 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 06:42:50.157455 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:42:50 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851370, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
Jun 19 06:42:50 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
I20260619 06:42:50.168219 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.155010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50354)
0619 06:42:50.155197 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:50.155216 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:50.155325 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:50.155618 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:50.155623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:50.155999 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:50.156192 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:50.156204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.157572 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.157578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:50.158696 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:50.158711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.159438 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.159444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:50.159658 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.160224 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:50.160246 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:50.162399 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:50.165283 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:50.165289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:50.165294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:50.165780 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:50.166157 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:50.166162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:50.166165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:50.166312 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:50.167027 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:50.167032 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:50.167359 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 06:42:50.167516 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 06:42:50.167834 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 06:42:50.168536 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.155230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.7.146.252:38411)
0619 06:42:50.155375 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:50.155380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:50.155393 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:42:50.155431 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:50.155434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:50.155488 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:50.155564 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:50.156337 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.157437 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.158844 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.159299 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.159397 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.162547 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:50.162552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:50.162555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:50.162582 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:50.165145 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.165875 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.165898 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.165902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.166027 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.166456 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.166462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.166465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.166847 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:50.166982 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:50.167645 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 06:42:50.168047 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":48}
I20260619 06:42:50.174399 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794: Bootstrap starting.
I20260619 06:42:50.175666 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.159610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.7.146.253:34251)
0619 06:42:50.159782 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:50.159787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:50.159804 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:42:50.159989 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:50.159994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:50.160068 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:50.160206 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:50.161408 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.162318 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.163695 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.164092 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.164206 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.169721 (+ 5515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:50.169727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:50.169732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:50.169774 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:50.172978 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.173670 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.173676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.173680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.173829 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.174209 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.174215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.174219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.174531 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:50.174820 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:50.175109 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 06:42:50.175340 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:42:50.178259 21381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:50.179530 21381 log.cc:824] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:50.180845 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:42:50.182873 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794: No bootstrap required, opened a new log
I20260619 06:42:50.193611 21381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } }
I20260619 06:42:50.194113 21381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:50.194247 21381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b25abd9c4d41a08d4398f7bdf76794, State: Initialized, Role: FOLLOWER
I20260619 06:42:50.195089 21381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } }
I20260619 06:42:50.197398 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } } }
I20260619 06:42:50.197861 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:42:50.198832 21381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:42:50.207333 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.186425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.7.146.253:34251)
0619 06:42:50.186605 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:50.186611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:50.187474 (+ 863us) server_negotiation.cc:407] Connection header received
0619 06:42:50.187738 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:50.187744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:50.187825 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:50.187893 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:50.190193 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.191312 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.194040 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.194734 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.194846 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.198837 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:50.198845 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:50.198850 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:50.198895 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:50.203416 (+ 4521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.204365 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.204371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.204374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.204544 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.204986 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.204991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.204995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.205381 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:50.206621 (+ 1240us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:50.206755 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 06:42:50.206985 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:42:50.209805 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.186422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.7.146.254:34335)
0619 06:42:50.186760 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:50.186765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:50.187721 (+ 956us) server_negotiation.cc:407] Connection header received
0619 06:42:50.187964 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:50.187970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:50.188048 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:50.188117 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:50.190497 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.191695 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.194281 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.194941 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.195031 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.199893 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:50.199900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:50.199905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:50.199953 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:50.203818 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.205044 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.205049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.205053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.205232 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.205756 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.205762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.205766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.206152 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:50.207355 (+ 1203us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:50.208672 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0619 06:42:50.208962 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260619 06:42:50.214159 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.187802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.7.146.252:38411)
0619 06:42:50.187999 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:50.188008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:50.194929 (+ 6921us) server_negotiation.cc:407] Connection header received
0619 06:42:50.195514 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:50.195520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:50.195605 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:50.195677 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:50.199563 (+ 3886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.200740 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.202512 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.203219 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.203339 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.206953 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:50.206959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:50.206963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:50.206999 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:50.210819 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.211568 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.211574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.211578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.211727 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.212099 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.212104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.212108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.212415 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:50.212764 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:50.213626 (+ 862us) server_negotiation.cc:299] Negotiation successful
0619 06:42:50.213842 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:42:50.219548 21405 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:42:50.219688 21405 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:42:50.215078 21314 master_runner.cc:186] Error getting master registration for 127.7.146.254:34335: OK, instance_id { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" instance_seqno: 1781851369139105 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:42:50.224429 21314 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:42:50.276110 21233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } attempt: 4
Jun 19 06:42:50 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851369, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
I20260619 06:42:50.290057 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.276794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.7.146.252:38411)
0619 06:42:50.276957 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:50.276966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:50.277021 (+ 55us) server_negotiation.cc:407] Connection header received
0619 06:42:50.277204 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:50.277209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:50.277294 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:50.277381 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:50.278177 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.278950 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.280278 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.280901 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.281018 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.284626 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:50.284634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:50.284639 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:50.284684 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:50.287390 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.288224 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.288228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.288231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.288381 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.288786 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.288789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.288791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.289068 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:50.289207 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:50.289627 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 06:42:50.289798 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:42:50.290206 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.276749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50382)
0619 06:42:50.276948 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:50.276968 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:50.277091 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:50.277423 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:50.277428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:50.277865 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:50.278021 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:50.278030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.279073 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.279080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:50.280013 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:50.280024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.281047 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.281053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:50.281239 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.281837 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:50.281855 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:50.284410 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:50.287544 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:50.287550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:50.287555 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:50.288091 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:50.288488 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:50.288493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:50.288497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:50.288695 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:50.289190 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:50.289195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:50.289388 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 06:42:50.289543 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 06:42:50.289827 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 06:42:50.294539 21233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147: Bootstrap starting.
I20260619 06:42:50.297683 21233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:50.299254 21233 log.cc:824] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:50.302075 21233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147: No bootstrap required, opened a new log
I20260619 06:42:50.309214 21233 raft_consensus.cc:348] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } }
I20260619 06:42:50.309553 21233 raft_consensus.cc:374] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:50.309676 21233 raft_consensus.cc:729] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e29ed3a9aa7f4132a7d8b5e610887147, State: Initialized, Role: FOLLOWER
I20260619 06:42:50.310256 21233 consensus_queue.cc:260] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } }
I20260619 06:42:50.311178 21408 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } } }
I20260619 06:42:50.311570 21408 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:42:50.312430 21233 sys_catalog.cc:564] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:42:50.323287 21419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:42:50.323415 21419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:42:50.322512 21166 master_runner.cc:186] Error getting master registration for 127.7.146.253:34251: OK, instance_id { permanent_uuid: "548eec022f844a1b994c707b437a8fda" instance_seqno: 1781851369617438 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:42:50.331537 21166 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 06:42:50.340415 21335 tablet.cc:2369] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:42:50.343364 21307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } attempt: 5
Jun 19 06:42:50 dist-test-slave-qfk4 krb5kdc[21153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851369, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
I20260619 06:42:50.358563 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.344115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:38411 (local address 127.0.0.1:50396)
0619 06:42:50.344750 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:50.344770 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:50.344884 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:50.346054 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:50.346060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:50.346628 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:50.346852 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:50.346866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.348187 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.348191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:50.349364 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:50.349379 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.350124 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.350130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:50.350337 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.350919 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:50.350937 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:50.353030 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:50.356045 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:50.356052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:50.356056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:50.356464 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:50.356877 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:50.356883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:50.356887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:50.357042 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:50.357606 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:50.357611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:50.357767 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 06:42:50.357883 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 06:42:50.358183 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502}
I20260619 06:42:50.358635 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:50.345368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.7.146.252:38411)
0619 06:42:50.345531 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:50.345536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:50.345554 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:42:50.345613 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:50.345617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:50.345693 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:50.346988 (+ 1295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:50.347182 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.348004 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.349509 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:50.349967 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:50.350085 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:50.353239 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:50.353245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:50.353249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:50.353281 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:50.355896 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.356630 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.356634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.356636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.356769 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:50.357188 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:50.357193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:50.357197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:50.357492 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:50.357647 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:50.358042 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 06:42:50.358282 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:42:50.363819 21307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda: Bootstrap starting.
I20260619 06:42:50.367798 21307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:50.368885 21307 log.cc:824] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:50.371461 21307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda: No bootstrap required, opened a new log
I20260619 06:42:50.378526 21307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } }
I20260619 06:42:50.378854 21307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:50.378958 21307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548eec022f844a1b994c707b437a8fda, State: Initialized, Role: FOLLOWER
W20260619 06:42:50.379410 21187 tablet.cc:2369] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:42:50.379705 21307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } }
I20260619 06:42:50.380738 21422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } } }
I20260619 06:42:50.381204 21422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:42:50.381975 21307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:42:50.401407 21240 master_runner.cc:263] No leader master found from master 548eec022f844a1b994c707b437a8fda
I20260619 06:42:50.401573 21240 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 06:42:50.410876 21433 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:42:50.411010 21433 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:42:50.462309 21394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:42:50.462677 21394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } }
I20260619 06:42:50.464440 21394 leader_election.cc:288] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e29ed3a9aa7f4132a7d8b5e610887147 (127.7.146.254:34335), 548eec022f844a1b994c707b437a8fda (127.7.146.253:34251)
I20260619 06:42:50.465724 21208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71b25abd9c4d41a08d4398f7bdf76794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" is_pre_election: true
I20260619 06:42:50.466367 21208 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71b25abd9c4d41a08d4398f7bdf76794 in term 0.
I20260619 06:42:50.467197 21333 leader_election.cc:302] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71b25abd9c4d41a08d4398f7bdf76794, e29ed3a9aa7f4132a7d8b5e610887147; no voters:
I20260619 06:42:50.468720 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71b25abd9c4d41a08d4398f7bdf76794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548eec022f844a1b994c707b437a8fda" is_pre_election: true
I20260619 06:42:50.469338 21282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71b25abd9c4d41a08d4398f7bdf76794 in term 0.
I20260619 06:42:50.470490 21394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:42:50.470664 21394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:42:50.470763 21394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:50.474464 21394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } }
I20260619 06:42:50.483722 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71b25abd9c4d41a08d4398f7bdf76794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548eec022f844a1b994c707b437a8fda"
I20260619 06:42:50.483992 21282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:50.487879 21282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71b25abd9c4d41a08d4398f7bdf76794 in term 1.
I20260619 06:42:50.488574 21331 leader_election.cc:302] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548eec022f844a1b994c707b437a8fda, 71b25abd9c4d41a08d4398f7bdf76794; no voters:
I20260619 06:42:50.489447 21394 leader_election.cc:288] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [CANDIDATE]: Term 1 election: Requested vote from peers e29ed3a9aa7f4132a7d8b5e610887147 (127.7.146.254:34335), 548eec022f844a1b994c707b437a8fda (127.7.146.253:34251)
I20260619 06:42:50.489863 21394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:42:50.490305 21394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [term 1 LEADER]: Becoming Leader. State: Replica: 71b25abd9c4d41a08d4398f7bdf76794, State: Running, Role: LEADER
I20260619 06:42:50.491164 21394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } }
I20260619 06:42:50.501168 21208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71b25abd9c4d41a08d4398f7bdf76794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e29ed3a9aa7f4132a7d8b5e610887147"
I20260619 06:42:50.501473 21208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:50.505220 21208 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71b25abd9c4d41a08d4398f7bdf76794 in term 1.
I20260619 06:42:50.514876 21435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71b25abd9c4d41a08d4398f7bdf76794. Latest consensus state: current_term: 1 leader_uuid: "71b25abd9c4d41a08d4398f7bdf76794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } } }
I20260619 06:42:50.515805 21435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:50.533249 21437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:42:50.535630 21437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:42:50.551235 21282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [term 1 FOLLOWER]: Refusing update from remote peer 71b25abd9c4d41a08d4398f7bdf76794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:42:50.552317 21435 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:42:50.559824 21208 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [term 1 FOLLOWER]: Refusing update from remote peer 71b25abd9c4d41a08d4398f7bdf76794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:42:50.560742 21435 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:42:50.584209 21408 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71b25abd9c4d41a08d4398f7bdf76794. Latest consensus state: current_term: 1 leader_uuid: "71b25abd9c4d41a08d4398f7bdf76794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } } }
I20260619 06:42:50.584591 21408 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:42:50.594169 21438 mvcc.cc:205] Tried to move back new op lower bound from 7298463213766508544 to 7298463213545205760. Current Snapshot: MvccSnapshot[applied={T|T < 7298463213766508544}]
I20260619 06:42:50.599000 21408 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71b25abd9c4d41a08d4398f7bdf76794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } } }
I20260619 06:42:50.599349 21408 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:42:50.600061 21435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71b25abd9c4d41a08d4398f7bdf76794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } } }
I20260619 06:42:50.600355 21435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:50.602126 21437 catalog_manager.cc:1347] Generated new cluster ID: 9117a5e257eb483890d3a69624c5aa03
I20260619 06:42:50.602237 21437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260619 06:42:50.607435 21261 tablet.cc:2369] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:42:50.617786 21422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71b25abd9c4d41a08d4398f7bdf76794. Latest consensus state: current_term: 1 leader_uuid: "71b25abd9c4d41a08d4398f7bdf76794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } } }
I20260619 06:42:50.618127 21422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:42:50.626396 21437 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:42:50.628189 21437 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:42:50.631803 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71b25abd9c4d41a08d4398f7bdf76794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } } }
I20260619 06:42:50.632114 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:50.634331 21439 mvcc.cc:205] Tried to move back new op lower bound from 7298463213766508544 to 7298463213545205760. Current Snapshot: MvccSnapshot[applied={T|T < 7298463213766508544}]
I20260619 06:42:50.641463 21437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71b25abd9c4d41a08d4398f7bdf76794: Generated new TSK 0
I20260619 06:42:50.642760 21437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:42:50.646188 21422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71b25abd9c4d41a08d4398f7bdf76794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e29ed3a9aa7f4132a7d8b5e610887147" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34335 } } peers { permanent_uuid: "548eec022f844a1b994c707b437a8fda" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 34251 } } peers { permanent_uuid: "71b25abd9c4d41a08d4398f7bdf76794" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 38411 } } }
I20260619 06:42:50.647949 21422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:42:51.325532 21419 catalog_manager.cc:1259] Loaded cluster ID: 9117a5e257eb483890d3a69624c5aa03
I20260619 06:42:51.325635 21419 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147: loading cluster ID for follower catalog manager: success
I20260619 06:42:51.328058 21419 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147: acquiring CA information for follower catalog manager: success
I20260619 06:42:51.329903 21419 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e29ed3a9aa7f4132a7d8b5e610887147: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:42:51.413062 21433 catalog_manager.cc:1259] Loaded cluster ID: 9117a5e257eb483890d3a69624c5aa03
I20260619 06:42:51.413160 21433 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda: loading cluster ID for follower catalog manager: success
I20260619 06:42:51.415797 21433 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda: acquiring CA information for follower catalog manager: success
I20260619 06:42:51.417690 21433 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 548eec022f844a1b994c707b437a8fda: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:42:51.790903 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 21166
I20260619 06:42:51.792181 21228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:51.900815 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 21166
I20260619 06:42:51.917248 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 21240
I20260619 06:42:51.918330 21302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:52.027408 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 21240
I20260619 06:42:52.043385 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 21314
I20260619 06:42:52.044456 21376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:52.151711 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 21314
2026-06-19T06:42:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4638 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:42:52 dist-test-slave-qfk4 krb5kdc[21457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:42:52 dist-test-slave-qfk4 krb5kdc[21457](info): set up 2 sockets
Jun 19 06:42:52 dist-test-slave-qfk4 krb5kdc[21457](info): commencing operation
krb5kdc: starting...
W20260619 06:42:53.164865 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:42:53 dist-test-slave-qfk4 krb5kdc[21457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:42:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:42:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:42:53.384357 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:39741
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:42:53.713920 21470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:53.714206 21470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:53.714332 21470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:53.721843 21470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:42:53.721976 21470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:53.722059 21470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:53.722127 21470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:42:53.722193 21470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:42:53.732172 21470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:53.733570 21470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:53.734874 21470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:53.745990 21475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:53.746033 21478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:53.746037 21476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:53.746858 21470 server_base.cc:1034] running on GCE node
I20260619 06:42:53.747535 21470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:53.748852 21470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:53.750057 21470 hybrid_clock.cc:648] HybridClock initialized: now 1781851373750012 us; error 30 us; skew 500 ppm
I20260619 06:42:53.750372 21470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:53 dist-test-slave-qfk4 krb5kdc[21457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851373, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:53.753334 21470 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:42:53.754743 21470 webserver.cc:460] Webserver started at http://127.7.146.254:42083/ using document root <none> and password file <none>
I20260619 06:42:53.755363 21470 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:53.755455 21470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:53.755707 21470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:42:53.758605 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "086487a20e054edea04b6f9126b10809"
format_stamp: "Formatted at 2026-06-19 06:42:53 on dist-test-slave-qfk4"
server_key: "9eedfeac108f5672c11094be2ef67919"
server_key_iv: "c85470a68925afb67f239954af72de2c"
server_key_version: "encryptionkey@0"
I20260619 06:42:53.759285 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "086487a20e054edea04b6f9126b10809"
format_stamp: "Formatted at 2026-06-19 06:42:53 on dist-test-slave-qfk4"
server_key: "9eedfeac108f5672c11094be2ef67919"
server_key_iv: "c85470a68925afb67f239954af72de2c"
server_key_version: "encryptionkey@0"
I20260619 06:42:53.764247 21470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:42:53.767625 21485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:53.768959 21470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:42:53.769105 21470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "086487a20e054edea04b6f9126b10809"
format_stamp: "Formatted at 2026-06-19 06:42:53 on dist-test-slave-qfk4"
server_key: "9eedfeac108f5672c11094be2ef67919"
server_key_iv: "c85470a68925afb67f239954af72de2c"
server_key_version: "encryptionkey@0"
I20260619 06:42:53.769287 21470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:53.783252 21470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:53.787372 21470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:53.787601 21470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:53.804343 21470 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:39741
I20260619 06:42:53.804363 21536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:39741 every 8 connection(s)
I20260619 06:42:53.805680 21470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:42:53.809360 21537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:42:53.815127 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 21470
I20260619 06:42:53.815326 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:42:53.815934 7755 external_mini_cluster.cc:1428] Setting key b4c7d4863aa57c58eb3abe9404dc5333
I20260619 06:42:53.819676 21537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809: Bootstrap starting.
I20260619 06:42:53.823231 21537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:53.824275 21537 log.cc:824] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:53.827200 21537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809: No bootstrap required, opened a new log
Jun 19 06:42:53 dist-test-slave-qfk4 krb5kdc[21457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:53.836968 21537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086487a20e054edea04b6f9126b10809" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39741 } }
I20260619 06:42:53.837446 21537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:53.837566 21537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086487a20e054edea04b6f9126b10809, State: Initialized, Role: FOLLOWER
I20260619 06:42:53.838238 21537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086487a20e054edea04b6f9126b10809" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39741 } }
I20260619 06:42:53.838523 21537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:42:53.838672 21537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:42:53.838842 21537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:53.839336 21540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:53.817672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.7.146.254:39741)
0619 06:42:53.818390 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:53.818401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:53.818440 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:42:53.820089 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:53.820113 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:53.820516 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:53.820868 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:53.822699 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:53.823993 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:53.825626 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:53.826274 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:53.826403 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:53.831082 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:53.831101 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:53.831120 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:53.831166 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:53.834938 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:53.835593 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:53.835602 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:53.835612 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:53.835811 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:53.836130 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:53.836135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:53.836138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:53.836862 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:53.837164 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:53.837523 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 06:42:53.837975 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":320,"threads_started":1}
I20260619 06:42:53.841351 21537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086487a20e054edea04b6f9126b10809" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39741 } }
I20260619 06:42:53.841854 21537 leader_election.cc:302] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 086487a20e054edea04b6f9126b10809; no voters:
I20260619 06:42:53.842494 21537 leader_election.cc:288] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:42:53.842594 21542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:42:53.843083 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:42:53.844913 21537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:42:53.844944 21542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [term 1 LEADER]: Becoming Leader. State: Replica: 086487a20e054edea04b6f9126b10809, State: Running, Role: LEADER
I20260619 06:42:53.845633 21542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086487a20e054edea04b6f9126b10809" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39741 } }
I20260619 06:42:53.848240 21543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "086487a20e054edea04b6f9126b10809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086487a20e054edea04b6f9126b10809" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39741 } } }
I20260619 06:42:53.848654 21543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:53.848875 21544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 086487a20e054edea04b6f9126b10809. Latest consensus state: current_term: 1 leader_uuid: "086487a20e054edea04b6f9126b10809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086487a20e054edea04b6f9126b10809" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 39741 } } }
I20260619 06:42:53.849202 21544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:53.853080 21554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:42:53.857102 21554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:42:53.864468 21554 catalog_manager.cc:1347] Generated new cluster ID: 150fe506776d4a7392255ae7957e425b
I20260619 06:42:53.864607 21554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:42:53.882704 21554 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:42:53.884616 21554 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:42:53.900389 21554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 086487a20e054edea04b6f9126b10809: Generated new TSK 0
I20260619 06:42:53.901497 21554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:42:53.955132 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 21470
I20260619 06:42:53.956332 21532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:54.052393 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 21470
2026-06-19T06:42:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1896 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:42:54 dist-test-slave-qfk4 krb5kdc[21565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:42:54 dist-test-slave-qfk4 krb5kdc[21565](info): set up 2 sockets
Jun 19 06:42:54 dist-test-slave-qfk4 krb5kdc[21565](info): commencing operation
krb5kdc: starting...
W20260619 06:42:55.068830 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:42:55 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:42:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:42:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:42:55.284474 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40389
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:40389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:42:55.600066 21578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:55.600378 21578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:55.600507 21578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:55.607793 21578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:42:55.607901 21578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:55.607955 21578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:55.608038 21578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:42:55.608088 21578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:42:55.618115 21578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:40389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:55.619406 21578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:55.620846 21578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:55.632194 21583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:55.632157 21586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:55.632247 21584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:55.633329 21578 server_base.cc:1034] running on GCE node
I20260619 06:42:55.634073 21578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:55.635426 21578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:55.636660 21578 hybrid_clock.cc:648] HybridClock initialized: now 1781851375636586 us; error 54 us; skew 500 ppm
I20260619 06:42:55.637004 21578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:55 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851375, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:55.643150 21578 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:42:55.644728 21578 webserver.cc:460] Webserver started at http://127.7.146.254:41481/ using document root <none> and password file <none>
I20260619 06:42:55.645407 21578 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:55.645531 21578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:55.645805 21578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:42:55.648782 21578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "ac522642c461494b8c6d5773380a5005"
format_stamp: "Formatted at 2026-06-19 06:42:55 on dist-test-slave-qfk4"
server_key: "9f0cba17d7c6c0ccf1dd25c835a9141e"
server_key_iv: "691cada2a2375e6b05f89f34cfe342fc"
server_key_version: "encryptionkey@0"
I20260619 06:42:55.649518 21578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "ac522642c461494b8c6d5773380a5005"
format_stamp: "Formatted at 2026-06-19 06:42:55 on dist-test-slave-qfk4"
server_key: "9f0cba17d7c6c0ccf1dd25c835a9141e"
server_key_iv: "691cada2a2375e6b05f89f34cfe342fc"
server_key_version: "encryptionkey@0"
I20260619 06:42:55.654587 21578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 06:42:55.658028 21593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:55.659420 21578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:42:55.659593 21578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "ac522642c461494b8c6d5773380a5005"
format_stamp: "Formatted at 2026-06-19 06:42:55 on dist-test-slave-qfk4"
server_key: "9f0cba17d7c6c0ccf1dd25c835a9141e"
server_key_iv: "691cada2a2375e6b05f89f34cfe342fc"
server_key_version: "encryptionkey@0"
I20260619 06:42:55.659773 21578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:55.681629 21578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:55.685940 21578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:55.686203 21578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:55.712591 21578 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:40389
I20260619 06:42:55.712591 21644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:40389 every 8 connection(s)
I20260619 06:42:55.713979 21578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:42:55.714861 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 21578
I20260619 06:42:55.715090 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:42:55.715963 7755 external_mini_cluster.cc:1428] Setting key b526903dfdeceae6dbf70fe21f833e34
I20260619 06:42:55.718106 21645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:42:55 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:55.733362 21645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Bootstrap starting.
I20260619 06:42:55.737828 21645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:55.739112 21645 log.cc:824] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:55.742280 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:55.718090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.7.146.254:40389)
0619 06:42:55.718954 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:55.718971 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:55.719021 (+ 50us) server_negotiation.cc:407] Connection header received
0619 06:42:55.721447 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:55.721479 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:55.722070 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:55.722575 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:55.724369 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:55.726381 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:55.728193 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:55.729263 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:55.729438 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:55.734716 (+ 5278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:55.734727 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:55.734745 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:55.734797 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:55.737866 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:55.738588 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:55.738605 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:55.738615 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:55.738802 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:55.739160 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:55.739165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:55.739169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:55.739770 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:55.740070 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:55.740545 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 06:42:55.740998 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":355,"threads_started":1}
I20260619 06:42:55.742398 21645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: No bootstrap required, opened a new log
I20260619 06:42:55.750216 21645 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:42:55.750576 21645 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:55.750671 21645 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac522642c461494b8c6d5773380a5005, State: Initialized, Role: FOLLOWER
I20260619 06:42:55.751224 21645 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:42:55.751439 21645 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:42:55.751612 21645 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:42:55.751762 21645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:55.753942 21645 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:42:55.754427 21645 leader_election.cc:302] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac522642c461494b8c6d5773380a5005; no voters:
I20260619 06:42:55.755045 21645 leader_election.cc:288] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:42:55.755218 21650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:42:55.756753 21650 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 1 LEADER]: Becoming Leader. State: Replica: ac522642c461494b8c6d5773380a5005, State: Running, Role: LEADER
I20260619 06:42:55.757298 21645 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:42:55.757428 21650 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:42:55.760722 21651 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac522642c461494b8c6d5773380a5005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } } }
I20260619 06:42:55.761089 21651 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:55.760476 21652 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac522642c461494b8c6d5773380a5005. Latest consensus state: current_term: 1 leader_uuid: "ac522642c461494b8c6d5773380a5005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } } }
I20260619 06:42:55.761663 21652 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: This master's current role is: LEADER
W20260619 06:42:55.773572 21666 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:42:55.773701 21666 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:42:55.773816 21659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:42:55.775337 21659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:42:55.783991 21659 catalog_manager.cc:1347] Generated new cluster ID: cbecd308bde742c68a14a13c9abfd94c
I20260619 06:42:55.784116 21659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:42:55.803546 21659 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:42:55.804881 21659 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:42:55.813036 21659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Generated new TSK 0
I20260619 06:42:55.813861 21659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:42:55.872807 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40389
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:42:56.197976 21671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:56.198242 21671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:56.198371 21671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:56.205523 21671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:56.205629 21671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:56.205777 21671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:42:56.215317 21671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:56.216612 21671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:56.217918 21671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:56.230651 21676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:56.230765 21679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:56.230800 21677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:56.231937 21671 server_base.cc:1034] running on GCE node
I20260619 06:42:56.232486 21671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:56.233403 21671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:56.234669 21671 hybrid_clock.cc:648] HybridClock initialized: now 1781851376234625 us; error 37 us; skew 500 ppm
I20260619 06:42:56.234956 21671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:56 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851376, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:56.237921 21671 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:42:56.239289 21671 webserver.cc:460] Webserver started at http://127.7.146.193:38825/ using document root <none> and password file <none>
I20260619 06:42:56.239915 21671 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:56.240006 21671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:56.240258 21671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:42:56.243101 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "2c00f0819a994c7fa1d68795889d3d7e"
format_stamp: "Formatted at 2026-06-19 06:42:56 on dist-test-slave-qfk4"
server_key: "189a14aabbf1bddcdf343457d2228cb1"
server_key_iv: "2b23276842f8235847809b56b50169c6"
server_key_version: "encryptionkey@0"
I20260619 06:42:56.243777 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "2c00f0819a994c7fa1d68795889d3d7e"
format_stamp: "Formatted at 2026-06-19 06:42:56 on dist-test-slave-qfk4"
server_key: "189a14aabbf1bddcdf343457d2228cb1"
server_key_iv: "2b23276842f8235847809b56b50169c6"
server_key_version: "encryptionkey@0"
I20260619 06:42:56.248692 21671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:42:56.251992 21686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:56.253407 21671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:42:56.253561 21671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "2c00f0819a994c7fa1d68795889d3d7e"
format_stamp: "Formatted at 2026-06-19 06:42:56 on dist-test-slave-qfk4"
server_key: "189a14aabbf1bddcdf343457d2228cb1"
server_key_iv: "2b23276842f8235847809b56b50169c6"
server_key_version: "encryptionkey@0"
I20260619 06:42:56.253741 21671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:56.276859 21671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:56.280784 21671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:56.281060 21671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:56.282063 21671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:56.283397 21671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:56.283483 21671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:56.283589 21671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:56.283640 21671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:56.313599 21671 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:44551
I20260619 06:42:56.313671 21798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:44551 every 8 connection(s)
I20260619 06:42:56.314895 21671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:42:56.316448 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 21671
I20260619 06:42:56.316658 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:42:56.317457 7755 external_mini_cluster.cc:1428] Setting key 32b03e8091db97f6f51e1e7df808a69b
Jun 19 06:42:56 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851376, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:56.337394 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:56.318265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:35325 (local address 127.7.146.254:40389)
0619 06:42:56.318705 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:56.318710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:56.320118 (+ 1408us) server_negotiation.cc:407] Connection header received
0619 06:42:56.322414 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:56.322418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:56.322486 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:56.322579 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:56.324528 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:56.325269 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:56.327306 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:56.327724 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:56.327845 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:56.331412 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:56.331419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:56.331424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:56.331464 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:56.334428 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:56.335106 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:56.335111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:56.335115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:56.335259 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:56.335623 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:56.335627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:56.335631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:56.335968 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:56.336116 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:56.336793 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 06:42:56.337017 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":234,"threads_started":1}
I20260619 06:42:56.338294 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:56.318885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:35325)
0619 06:42:56.319728 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:56.319778 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:56.322138 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:56.322840 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:56.322851 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:56.323553 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:56.324356 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:56.324385 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:56.325415 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:56.325421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:56.327169 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:56.327183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:56.327946 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:56.327951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:56.328151 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:56.328886 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:56.328909 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:56.331167 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:56.334561 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:56.334570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:56.334585 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:56.334998 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:56.335353 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:56.335356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:56.335358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:56.335524 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:56.336085 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:56.336091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:56.336401 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 06:42:56.336664 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 06:42:56.337068 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":325,"threads_started":1}
I20260619 06:42:56.339771 21799 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40389
I20260619 06:42:56.340106 21799 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:56.341029 21799 heartbeater.cc:511] Master 127.7.146.254:40389 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
I20260619 06:42:56.343259 21610 ts_manager.cc:194] Registered new tserver with Master: 2c00f0819a994c7fa1d68795889d3d7e (127.7.146.193:44551)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
I20260619 06:42:56.345163 21610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:35325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:42:56.369511 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40389
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:42:56.685592 21805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:56.685876 21805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:56.686003 21805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:56.693449 21805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:56.693576 21805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:56.693768 21805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:42:56.703719 21805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:56.705049 21805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:56.706391 21805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:56.721304 21811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:56.721336 21810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:56.722020 21813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:56.729839 21805 server_base.cc:1034] running on GCE node
I20260619 06:42:56.730499 21805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:56.731528 21805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:56.732744 21805 hybrid_clock.cc:648] HybridClock initialized: now 1781851376732690 us; error 44 us; skew 500 ppm
I20260619 06:42:56.733088 21805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:56 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851376, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:56.736275 21805 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:42:56.737902 21805 webserver.cc:460] Webserver started at http://127.7.146.194:35447/ using document root <none> and password file <none>
I20260619 06:42:56.738684 21805 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:56.738791 21805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:56.739082 21805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:42:56.742003 21805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "3c98a2a5f0f4482da42b0cd37347fe1b"
format_stamp: "Formatted at 2026-06-19 06:42:56 on dist-test-slave-qfk4"
server_key: "7c0a8cd576ea2a55ae69f1be975a7613"
server_key_iv: "191447ed0eb06bf52d990253ec7d7773"
server_key_version: "encryptionkey@0"
I20260619 06:42:56.742748 21805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "3c98a2a5f0f4482da42b0cd37347fe1b"
format_stamp: "Formatted at 2026-06-19 06:42:56 on dist-test-slave-qfk4"
server_key: "7c0a8cd576ea2a55ae69f1be975a7613"
server_key_iv: "191447ed0eb06bf52d990253ec7d7773"
server_key_version: "encryptionkey@0"
I20260619 06:42:56.747828 21805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 06:42:56.751291 21820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:56.752874 21805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:42:56.753050 21805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "3c98a2a5f0f4482da42b0cd37347fe1b"
format_stamp: "Formatted at 2026-06-19 06:42:56 on dist-test-slave-qfk4"
server_key: "7c0a8cd576ea2a55ae69f1be975a7613"
server_key_iv: "191447ed0eb06bf52d990253ec7d7773"
server_key_version: "encryptionkey@0"
I20260619 06:42:56.753273 21805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:56.776614 21805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:56.780535 21805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:56.780814 21805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:56.781791 21805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:56.783072 21805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:56.783144 21805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:56.783257 21805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:56.783303 21805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:56.812755 21805 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:40867
I20260619 06:42:56.812794 21932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:40867 every 8 connection(s)
I20260619 06:42:56.814100 21805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:42:56.820187 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 21805
I20260619 06:42:56.820374 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:42:56.820950 7755 external_mini_cluster.cc:1428] Setting key 5620a6ff5cc0007f8443db94bd705c39
Jun 19 06:42:56 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851376, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:56.836216 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:56.816520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:54471 (local address 127.7.146.254:40389)
0619 06:42:56.816730 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:56.816735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:56.817870 (+ 1135us) server_negotiation.cc:407] Connection header received
0619 06:42:56.819628 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:56.819632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:56.819701 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:56.819789 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:56.821831 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:56.822812 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:56.824968 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:56.825467 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:56.825578 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:56.829707 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:56.829713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:56.829717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:56.829755 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:56.833059 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:56.833857 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:56.833862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:56.833866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:56.834025 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:56.834415 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:56.834419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:56.834422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:56.834779 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:56.834921 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:56.835636 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 06:42:56.835875 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 06:42:56.837127 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:56.817023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:54471)
0619 06:42:56.817708 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:56.817749 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:56.819370 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:56.820068 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:56.820079 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:56.820860 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:56.821644 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:56.821664 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:56.822952 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:56.822956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:56.824794 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:56.824809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:56.825683 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:56.825690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:56.825948 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:56.826705 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:56.826733 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:56.829419 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:56.833227 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:56.833235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:56.833250 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:56.833722 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:56.834124 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:56.834127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:56.834130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:56.834310 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:56.834905 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:56.834912 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:56.835267 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 06:42:56.835503 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 06:42:56.835916 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":245,"threads_started":1}
I20260619 06:42:56.838606 21933 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40389
I20260619 06:42:56.838902 21933 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:56.839619 21933 heartbeater.cc:511] Master 127.7.146.254:40389 requested a full tablet report, sending...
I20260619 06:42:56.841163 21610 ts_manager.cc:194] Registered new tserver with Master: 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867)
I20260619 06:42:56.841988 21610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:54471
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:42:56.871181 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40389
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:42:57.186002 21938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:57.186300 21938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:57.186419 21938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:57.193954 21938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:57.194098 21938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:57.194254 21938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:42:57.204279 21938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:57.205657 21938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:57.206993 21938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:57.219926 21944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:57.220026 21946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:57.220471 21943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:57.220983 21938 server_base.cc:1034] running on GCE node
I20260619 06:42:57.221601 21938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:57.222560 21938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:57.223796 21938 hybrid_clock.cc:648] HybridClock initialized: now 1781851377223735 us; error 45 us; skew 500 ppm
I20260619 06:42:57.224170 21938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:57 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851377, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:57.227540 21938 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:42:57.229161 21938 webserver.cc:460] Webserver started at http://127.7.146.195:40245/ using document root <none> and password file <none>
I20260619 06:42:57.229945 21938 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:57.230048 21938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:57.230332 21938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:42:57.233276 21938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "2ff3edebbbb145d89b15575d6c8ee0dd"
format_stamp: "Formatted at 2026-06-19 06:42:57 on dist-test-slave-qfk4"
server_key: "8c6c0c6273686c8f7d4e5e35e4723efc"
server_key_iv: "e66aabd6114503ad6ef706c5448ae458"
server_key_version: "encryptionkey@0"
I20260619 06:42:57.234014 21938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "2ff3edebbbb145d89b15575d6c8ee0dd"
format_stamp: "Formatted at 2026-06-19 06:42:57 on dist-test-slave-qfk4"
server_key: "8c6c0c6273686c8f7d4e5e35e4723efc"
server_key_iv: "e66aabd6114503ad6ef706c5448ae458"
server_key_version: "encryptionkey@0"
I20260619 06:42:57.239418 21938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:42:57.243407 21953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:57.244944 21938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:42:57.245102 21938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "2ff3edebbbb145d89b15575d6c8ee0dd"
format_stamp: "Formatted at 2026-06-19 06:42:57 on dist-test-slave-qfk4"
server_key: "8c6c0c6273686c8f7d4e5e35e4723efc"
server_key_iv: "e66aabd6114503ad6ef706c5448ae458"
server_key_version: "encryptionkey@0"
I20260619 06:42:57.245293 21938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:57.263387 21938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:57.267829 21938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:57.268064 21938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:57.269008 21938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:57.270408 21938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:57.270479 21938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:57.270591 21938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:57.270632 21938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:57.300354 21938 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:42377
I20260619 06:42:57.300388 22065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:42377 every 8 connection(s)
I20260619 06:42:57.301694 21938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:42:57.311867 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 21938
I20260619 06:42:57.312049 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:42:57.312640 7755 external_mini_cluster.cc:1428] Setting key a6462648594246a55764741fce5814d6
Jun 19 06:42:57 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851377, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:57.321898 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.304271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:52813 (local address 127.7.146.254:40389)
0619 06:42:57.304453 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:57.304458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:57.305730 (+ 1272us) server_negotiation.cc:407] Connection header received
0619 06:42:57.307509 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:57.307514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:57.307616 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:57.307704 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:57.309225 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.309938 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.311492 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.311900 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.312021 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.316205 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:57.316211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:57.316214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:57.316251 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:57.318897 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:57.319577 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:57.319580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:57.319583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:57.319742 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:57.320198 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:57.320203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:57.320205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:57.320602 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:57.320742 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:57.321386 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 06:42:57.321627 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:42:57.322816 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.304850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:52813)
0619 06:42:57.305560 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:57.305603 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:57.307266 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:57.307908 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:57.307917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:57.308423 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:57.309082 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.309099 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.310070 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.310074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.311340 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.311352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.312154 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.312162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.312441 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.313257 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:57.313294 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:57.315970 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:57.319033 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:57.319040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:57.319053 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:57.319476 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:57.319835 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:57.319839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:57.319841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:57.320036 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:57.320731 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:57.320738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:57.321049 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 06:42:57.321272 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 06:42:57.321675 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":275,"threads_started":1}
I20260619 06:42:57.324412 22066 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40389
I20260619 06:42:57.324780 22066 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:57.325541 22066 heartbeater.cc:511] Master 127.7.146.254:40389 requested a full tablet report, sending...
I20260619 06:42:57.327088 21610 ts_manager.cc:194] Registered new tserver with Master: 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377)
I20260619 06:42:57.328119 21610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:52813
I20260619 06:42:57.328375 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:42:57.344328 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.332798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.7.146.254:40389)
0619 06:42:57.332955 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:57.332959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:57.333340 (+ 381us) server_negotiation.cc:407] Connection header received
0619 06:42:57.333493 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:57.333496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:57.333557 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:57.333636 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:57.334872 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.335547 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.336882 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.337271 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.337359 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.339683 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:57.339689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:57.339692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:57.339725 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:57.342169 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:57.342787 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:57.342791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:57.342793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:57.342919 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:57.343177 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:57.343180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:57.343182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:57.343480 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:57.343657 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:57.343923 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 06:42:57.344091 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:42:57.348140 21799 heartbeater.cc:503] Master 127.7.146.254:40389 was elected leader, sending a full tablet report...
I20260619 06:42:57.348106 21610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:42:57.355294 21610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:42:57.403559 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.367556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:40867 (local address 127.0.0.1:49144)
0619 06:42:57.392820 (+ 25264us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:57.392841 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:57.392977 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:57.394919 (+ 1942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:57.394929 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:57.394974 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:42:57.396471 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.396492 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.398441 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.398446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.400795 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.400808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.402585 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.402591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.402770 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.402827 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 06:42:57.402961 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 06:42:57.403158 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25117,"mutex_wait_us":144,"thread_start_us":28771,"threads_started":1}
I20260619 06:42:57.404162 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.367801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.7.146.195:42377)
0619 06:42:57.395951 (+ 28150us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:57.395962 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:57.396000 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:42:57.396111 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:57.396119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:57.396394 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:57.396664 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:42:57.397236 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.398527 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.401179 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.402474 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.402576 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.402785 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:57.403670 (+ 885us) server_negotiation.cc:299] Negotiation successful
0619 06:42:57.403838 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27951,"thread_start_us":27755,"threads_started":1}
I20260619 06:42:57.404161 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.369468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.7.146.193:44551)
0619 06:42:57.392109 (+ 22641us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:57.392119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:57.397021 (+ 4902us) server_negotiation.cc:407] Connection header received
0619 06:42:57.397149 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:57.397157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:57.397388 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:57.397594 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:42:57.398122 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.399546 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.401719 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.402734 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.402862 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.403082 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:57.403504 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 06:42:57.403675 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22438,"thread_start_us":22675,"threads_started":1}
I20260619 06:42:57.404259 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.367779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.7.146.194:40867)
0619 06:42:57.394098 (+ 26319us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:57.394109 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:57.394143 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:42:57.394262 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:57.394271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:57.394588 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:57.394844 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:42:57.396864 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.398305 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.400957 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.402365 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.402512 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.402748 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:57.403826 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0619 06:42:57.403989 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26117,"thread_start_us":27258,"threads_started":1}
I20260619 06:42:57.403966 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.369254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.0.0.1:55054)
0619 06:42:57.396731 (+ 27477us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:57.396752 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:57.396876 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:57.397591 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:57.397601 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:57.397628 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:42:57.397957 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.397969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.399719 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.399727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.401586 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.401597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.402992 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.402997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.403265 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.403326 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 06:42:57.403414 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:42:57.403561 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27283,"mutex_wait_us":58,"thread_start_us":19494,"threads_started":1}
I20260619 06:42:57.404301 22082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.367486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:42377 (local address 127.0.0.1:35132)
0619 06:42:57.389183 (+ 21697us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:57.389232 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:57.389408 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:57.396635 (+ 7227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:57.396640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:57.396668 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:42:57.397036 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.397046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.398671 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.398677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.401005 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.401020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.402980 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.402987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.403273 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.403332 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 06:42:57.403414 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:42:57.403561 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21411,"mutex_wait_us":95,"thread_start_us":25060,"threads_started":1}
I20260619 06:42:57.407976 22001 tablet_service.cc:1467] Processing CreateTablet for tablet f9c503daa6014e2ea5defd47af735c96 (DEFAULT_TABLE table=test-table [id=ed69b78fc1d342f1b35d5e76df60c633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:42:57.408115 21868 tablet_service.cc:1467] Processing CreateTablet for tablet f9c503daa6014e2ea5defd47af735c96 (DEFAULT_TABLE table=test-table [id=ed69b78fc1d342f1b35d5e76df60c633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:42:57.408115 21734 tablet_service.cc:1467] Processing CreateTablet for tablet f9c503daa6014e2ea5defd47af735c96 (DEFAULT_TABLE table=test-table [id=ed69b78fc1d342f1b35d5e76df60c633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:42:57.409466 22001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c503daa6014e2ea5defd47af735c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:42:57.409487 21734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c503daa6014e2ea5defd47af735c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:42:57.409482 21868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c503daa6014e2ea5defd47af735c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:42:57.420861 22084 tablet_bootstrap.cc:492] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: Bootstrap starting.
I20260619 06:42:57.424103 22084 tablet_bootstrap.cc:654] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:57.425032 22085 tablet_bootstrap.cc:492] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: Bootstrap starting.
I20260619 06:42:57.425515 22084 log.cc:824] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:57.425904 22086 tablet_bootstrap.cc:492] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: Bootstrap starting.
I20260619 06:42:57.428685 22084 tablet_bootstrap.cc:492] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: No bootstrap required, opened a new log
I20260619 06:42:57.428966 22084 ts_tablet_manager.cc:1397] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260619 06:42:57.429701 22086 tablet_bootstrap.cc:654] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:57.429780 22085 tablet_bootstrap.cc:654] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: Neither blocks nor log segments found. Creating new log.
I20260619 06:42:57.431021 22086 log.cc:824] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:57.431012 22085 log.cc:824] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:57.435365 22085 tablet_bootstrap.cc:492] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: No bootstrap required, opened a new log
I20260619 06:42:57.435365 22086 tablet_bootstrap.cc:492] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: No bootstrap required, opened a new log
I20260619 06:42:57.435648 22085 ts_tablet_manager.cc:1397] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260619 06:42:57.435648 22086 ts_tablet_manager.cc:1397] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 06:42:57.436973 22084 raft_consensus.cc:348] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } } peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } }
I20260619 06:42:57.437335 22084 raft_consensus.cc:374] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:57.437456 22084 raft_consensus.cc:729] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff3edebbbb145d89b15575d6c8ee0dd, State: Initialized, Role: FOLLOWER
I20260619 06:42:57.438035 22084 consensus_queue.cc:260] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } } peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } }
I20260619 06:42:57.439055 22066 heartbeater.cc:503] Master 127.7.146.254:40389 was elected leader, sending a full tablet report...
I20260619 06:42:57.439754 22084 ts_tablet_manager.cc:1428] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260619 06:42:57.446138 22085 raft_consensus.cc:348] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } } peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } }
I20260619 06:42:57.446138 22086 raft_consensus.cc:348] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } } peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } }
I20260619 06:42:57.446626 22086 raft_consensus.cc:374] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:57.446626 22085 raft_consensus.cc:374] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:42:57.446735 22085 raft_consensus.cc:729] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c00f0819a994c7fa1d68795889d3d7e, State: Initialized, Role: FOLLOWER
I20260619 06:42:57.446731 22086 raft_consensus.cc:729] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c98a2a5f0f4482da42b0cd37347fe1b, State: Initialized, Role: FOLLOWER
I20260619 06:42:57.447378 22086 consensus_queue.cc:260] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } } peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } }
I20260619 06:42:57.447378 22085 consensus_queue.cc:260] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } } peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } }
I20260619 06:42:57.449326 22085 ts_tablet_manager.cc:1428] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260619 06:42:57.449728 21933 heartbeater.cc:503] Master 127.7.146.254:40389 was elected leader, sending a full tablet report...
I20260619 06:42:57.450343 22086 ts_tablet_manager.cc:1428] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260619 06:42:57.553282 22067 tablet.cc:2369] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:42:57.557991 22092 raft_consensus.cc:482] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:42:57.558327 22092 raft_consensus.cc:504] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } } peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } }
I20260619 06:42:57.560176 22092 leader_election.cc:288] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377), 2c00f0819a994c7fa1d68795889d3d7e (127.7.146.193:44551)
W20260619 06:42:57.566010 21934 tablet.cc:2369] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:42:57.567113 21800 tablet.cc:2369] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:42:57.568141 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.560982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:58477 (local address 127.7.146.195:42377)
0619 06:42:57.561158 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:57.561163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:57.561923 (+ 760us) server_negotiation.cc:407] Connection header received
0619 06:42:57.562081 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:57.562087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:57.562169 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:57.562263 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:42:57.562846 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.563849 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.566343 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.567288 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.567435 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.567510 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:57.567777 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 06:42:57.567902 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:42:57.568282 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.560591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:42377 (local address 127.7.146.194:58477)
0619 06:42:57.561788 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:57.561807 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:57.561957 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:57.562299 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:57.562304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:57.562331 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:42:57.562723 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.562735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.563959 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.563963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.566141 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.566156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.567484 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.567490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.567632 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.567656 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:42:57.567736 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:42:57.567860 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":3,"thread_start_us":1517,"threads_started":1}
I20260619 06:42:57.568689 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.560909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:59303 (local address 127.7.146.193:44551)
0619 06:42:57.561103 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:57.561108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:57.562560 (+ 1452us) server_negotiation.cc:407] Connection header received
0619 06:42:57.562746 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:57.562751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:57.562826 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:57.562931 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:42:57.563358 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.564288 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.566557 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.567658 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.567810 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.567879 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:57.568328 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 06:42:57.568456 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:42:57.569254 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.560598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.7.146.194:59303)
0619 06:42:57.562453 (+ 1855us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:57.562473 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:57.562588 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:57.562935 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:57.562938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:57.562957 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:42:57.563233 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.563240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.564418 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.564423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.566416 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:57.566427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.568017 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.568022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:57.568153 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.568175 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:42:57.568906 (+ 731us) client_negotiation.cc:240] Negotiation successful
0619 06:42:57.568999 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1735,"mutex_wait_us":45,"thread_start_us":848,"threads_started":1}
I20260619 06:42:57.569273 22021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c503daa6014e2ea5defd47af735c96" candidate_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" is_pre_election: true
I20260619 06:42:57.569782 22021 raft_consensus.cc:2454] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c98a2a5f0f4482da42b0cd37347fe1b in term 0.
I20260619 06:42:57.570218 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c503daa6014e2ea5defd47af735c96" candidate_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c00f0819a994c7fa1d68795889d3d7e" is_pre_election: true
I20260619 06:42:57.570467 21824 leader_election.cc:302] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff3edebbbb145d89b15575d6c8ee0dd, 3c98a2a5f0f4482da42b0cd37347fe1b; no voters:
I20260619 06:42:57.570739 21754 raft_consensus.cc:2454] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c98a2a5f0f4482da42b0cd37347fe1b in term 0.
I20260619 06:42:57.570860 22092 raft_consensus.cc:2781] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:42:57.571002 22092 raft_consensus.cc:482] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:42:57.571086 22092 raft_consensus.cc:3037] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:57.573277 22092 raft_consensus.cc:504] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } } peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } }
I20260619 06:42:57.573828 22092 leader_election.cc:288] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [CANDIDATE]: Term 1 election: Requested vote from peers 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377), 2c00f0819a994c7fa1d68795889d3d7e (127.7.146.193:44551)
I20260619 06:42:57.574261 22021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c503daa6014e2ea5defd47af735c96" candidate_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd"
I20260619 06:42:57.574271 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c503daa6014e2ea5defd47af735c96" candidate_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c00f0819a994c7fa1d68795889d3d7e"
I20260619 06:42:57.574466 21754 raft_consensus.cc:3037] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:57.574465 22021 raft_consensus.cc:3037] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:42:57.577570 21754 raft_consensus.cc:2454] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c98a2a5f0f4482da42b0cd37347fe1b in term 1.
I20260619 06:42:57.577560 22021 raft_consensus.cc:2454] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c98a2a5f0f4482da42b0cd37347fe1b in term 1.
I20260619 06:42:57.577975 21824 leader_election.cc:302] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c00f0819a994c7fa1d68795889d3d7e, 2ff3edebbbb145d89b15575d6c8ee0dd, 3c98a2a5f0f4482da42b0cd37347fe1b; no voters:
I20260619 06:42:57.578315 22092 raft_consensus.cc:2781] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:42:57.578728 22092 raft_consensus.cc:686] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 1 LEADER]: Becoming Leader. State: Replica: 3c98a2a5f0f4482da42b0cd37347fe1b, State: Running, Role: LEADER
I20260619 06:42:57.579178 22092 consensus_queue.cc:237] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } } peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } }
I20260619 06:42:57.583463 21609 catalog_manager.cc:5495] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194). New cstate: current_term: 1 leader_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } health_report { overall_health: UNKNOWN } } }
I20260619 06:42:57.618721 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.613770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.7.146.194:40867)
0619 06:42:57.613950 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:57.613954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:57.613971 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:42:57.614081 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:57.614085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:57.614176 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:57.614293 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:42:57.614580 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.615413 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.616789 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.617205 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.617301 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.617709 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:42:57.618227 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:42:57.618331 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:57.618374 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 06:42:57.618439 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:42:57.626569 21754 raft_consensus.cc:1261] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Refusing update from remote peer 3c98a2a5f0f4482da42b0cd37347fe1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:42:57.626567 22021 raft_consensus.cc:1261] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 1 FOLLOWER]: Refusing update from remote peer 3c98a2a5f0f4482da42b0cd37347fe1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:42:57.627429 22095 consensus_queue.cc:1035] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:42:57.627861 22092 consensus_queue.cc:1035] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:42:57.638190 22100 mvcc.cc:205] Tried to move back new op lower bound from 7298463242748997632 to 7298463242568089600. Current Snapshot: MvccSnapshot[applied={T|T < 7298463242748997632}]
I20260619 06:42:57.653554 21609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:42:57.653867 21609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:42:57.657549 21609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Sending DeleteTablet for 3 replicas of tablet f9c503daa6014e2ea5defd47af735c96
I20260619 06:42:57.658741 22001 tablet_service.cc:1514] Processing DeleteTablet for tablet f9c503daa6014e2ea5defd47af735c96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:42:57 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:35132
I20260619 06:42:57.658766 21734 tablet_service.cc:1514] Processing DeleteTablet for tablet f9c503daa6014e2ea5defd47af735c96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:42:57 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:55054
I20260619 06:42:57.659641 22109 tablet_replica.cc:330] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: stopping tablet replica
I20260619 06:42:57.659616 21868 tablet_service.cc:1514] Processing DeleteTablet for tablet f9c503daa6014e2ea5defd47af735c96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:42:57 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:49144
I20260619 06:42:57.660138 22109 raft_consensus.cc:2229] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:42:57.660748 22109 raft_consensus.cc:2258] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:42:57.660796 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 21671
I20260619 06:42:57.663074 22109 ts_tablet_manager.cc:1905] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:42:57.670848 22109 ts_tablet_manager.cc:1918] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:42:57.671126 22109 log.cc:1196] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/f9c503daa6014e2ea5defd47af735c96
I20260619 06:42:57.671689 22109 ts_tablet_manager.cc:1939] T f9c503daa6014e2ea5defd47af735c96 P 2c00f0819a994c7fa1d68795889d3d7e: Deleting consensus metadata
I20260619 06:42:57.673835 21594 catalog_manager.cc:4841] TS 2c00f0819a994c7fa1d68795889d3d7e (127.7.146.193:44551): tablet f9c503daa6014e2ea5defd47af735c96 (table test-table [id=ed69b78fc1d342f1b35d5e76df60c633]) successfully deleted
I20260619 06:42:57.674163 22108 tablet_replica.cc:330] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: stopping tablet replica
I20260619 06:42:57.674558 22108 raft_consensus.cc:2229] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:42:57.675016 22108 raft_consensus.cc:2258] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:42:57.676940 22108 ts_tablet_manager.cc:1905] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:42:57.677889 22110 tablet_replica.cc:330] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: stopping tablet replica
I20260619 06:42:57.678371 22110 raft_consensus.cc:2229] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:42:57.679078 22110 raft_consensus.cc:2258] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:42:57.681641 22110 ts_tablet_manager.cc:1905] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:42:57.683758 22108 ts_tablet_manager.cc:1918] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:42:57.683975 22108 log.cc:1196] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/f9c503daa6014e2ea5defd47af735c96
I20260619 06:42:57.684537 22108 ts_tablet_manager.cc:1939] T f9c503daa6014e2ea5defd47af735c96 P 2ff3edebbbb145d89b15575d6c8ee0dd: Deleting consensus metadata
I20260619 06:42:57.686803 21597 catalog_manager.cc:4841] TS 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377): tablet f9c503daa6014e2ea5defd47af735c96 (table test-table [id=ed69b78fc1d342f1b35d5e76df60c633]) successfully deleted
Jun 19 06:42:57 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:42:57.688731 22110 ts_tablet_manager.cc:1918] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:42:57.688894 22110 log.cc:1196] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/f9c503daa6014e2ea5defd47af735c96
I20260619 06:42:57.689347 22110 ts_tablet_manager.cc:1939] T f9c503daa6014e2ea5defd47af735c96 P 3c98a2a5f0f4482da42b0cd37347fe1b: Deleting consensus metadata
I20260619 06:42:57.691216 21595 catalog_manager.cc:4841] TS 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867): tablet f9c503daa6014e2ea5defd47af735c96 (table test-table [id=ed69b78fc1d342f1b35d5e76df60c633]) successfully deleted
I20260619 06:42:57.695622 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.662067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.7.146.193:44551)
0619 06:42:57.662247 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:57.662253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:57.677187 (+ 14934us) server_negotiation.cc:407] Connection header received
0619 06:42:57.677388 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:57.677394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:57.677483 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:57.677588 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:57.679762 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.680955 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.683001 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.683665 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.683772 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.689493 (+ 5721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:57.689513 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:57.689531 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:57.689586 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:57.692869 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:57.693747 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:57.693755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:57.693757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:57.693897 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:57.694290 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:57.694293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:57.694295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:57.694644 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:57.694865 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:57.695188 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 06:42:57.695355 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:42:57.696362 21794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:57.788605 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 21671
I20260619 06:42:57.809121 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 21805
Jun 19 06:42:57 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:42:57.828928 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.810059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.7.146.194:40867)
0619 06:42:57.810254 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:57.810260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:57.810280 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:42:57.810344 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:57.810348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:57.810432 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:57.810514 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:57.811757 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.812971 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.814910 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.815645 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.815777 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.821340 (+ 5563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:57.821356 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:57.821375 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:57.821431 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:57.825715 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:57.826596 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:57.826606 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:57.826609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:57.826785 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:57.827202 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:57.827208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:57.827211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:57.827686 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:57.828014 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:57.828287 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 06:42:57.828510 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:42:57.830049 21928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:57.942191 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 21805
I20260619 06:42:57.956283 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 21938
Jun 19 06:42:57 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:42:57.971478 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:57.957047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.7.146.195:42377)
0619 06:42:57.957224 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:57.957230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:57.957272 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:42:57.957481 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:57.957485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:57.957565 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:57.957657 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:57.958980 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.959679 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.961031 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:57.961398 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:57.961504 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:57.965807 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:57.965822 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:57.965839 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:57.965875 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:57.968775 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:57.969485 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:57.969493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:57.969495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:57.969636 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:57.969994 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:57.970000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:57.970003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:57.970335 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:57.970602 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:57.970933 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 06:42:57.971146 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:42:57.972128 22061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:58.059890 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 21938
I20260619 06:42:58.072790 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 21578
I20260619 06:42:58.073899 21640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:42:58.184688 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 21578
I20260619 06:42:58.201536 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40389
--webserver_interface=127.7.146.254
--webserver_port=41481
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:40389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:42:58.522951 22116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:58.523236 22116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:58.523330 22116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:58.530563 22116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:42:58.530668 22116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:58.530719 22116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:58.530773 22116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:42:58.530817 22116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:42:58.540756 22116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:40389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:58.541978 22116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:58.543221 22116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:58.564049 22123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:58.564046 22122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:58.564039 22125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:58.564947 22116 server_base.cc:1034] running on GCE node
I20260619 06:42:58.565841 22116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:58.567449 22116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:58.568701 22116 hybrid_clock.cc:648] HybridClock initialized: now 1781851378568621 us; error 58 us; skew 500 ppm
I20260619 06:42:58.569093 22116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:58 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851378, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:58.572484 22116 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:42:58.574272 22116 webserver.cc:460] Webserver started at http://127.7.146.254:41481/ using document root <none> and password file <none>
I20260619 06:42:58.575031 22116 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:58.575165 22116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:58.581398 22116 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:42:58.584393 22132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:58.585922 22116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:42:58.586088 22116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "ac522642c461494b8c6d5773380a5005"
format_stamp: "Formatted at 2026-06-19 06:42:55 on dist-test-slave-qfk4"
server_key: "9f0cba17d7c6c0ccf1dd25c835a9141e"
server_key_iv: "691cada2a2375e6b05f89f34cfe342fc"
server_key_version: "encryptionkey@0"
I20260619 06:42:58.587174 22116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:58.606204 22116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:58.610633 22116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:58.610905 22116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:58.628329 22116 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:40389
I20260619 06:42:58.628340 22183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:40389 every 8 connection(s)
I20260619 06:42:58.629941 22116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:42:58.631548 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 22116
I20260619 06:42:58.632279 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:44551
--local_ip_for_outbound_sockets=127.7.146.193
--tserver_master_addrs=127.7.146.254:40389
--webserver_port=38825
--webserver_interface=127.7.146.193
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:42:58.637400 22184 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:42:58.641993 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Bootstrap starting.
I20260619 06:42:58.670171 22184 log.cc:824] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Log is configured to *not* fsync() on all Append() calls
I20260619 06:42:58.686247 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:42:58.686941 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Bootstrap complete.
I20260619 06:42:58.697120 22184 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:42:58.698040 22184 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac522642c461494b8c6d5773380a5005, State: Initialized, Role: FOLLOWER
I20260619 06:42:58.698727 22184 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:42:58.698966 22184 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:42:58.699077 22184 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:42:58.699237 22184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:42:58.703286 22184 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:42:58.703826 22184 leader_election.cc:302] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac522642c461494b8c6d5773380a5005; no voters:
I20260619 06:42:58.704392 22184 leader_election.cc:288] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:42:58.704635 22188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:42:58.705588 22188 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 2 LEADER]: Becoming Leader. State: Replica: ac522642c461494b8c6d5773380a5005, State: Running, Role: LEADER
I20260619 06:42:58.706235 22188 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:42:58.706437 22184 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:42:58.708647 22189 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ac522642c461494b8c6d5773380a5005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } } }
I20260619 06:42:58.709028 22189 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:58.713613 22190 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac522642c461494b8c6d5773380a5005. Latest consensus state: current_term: 2 leader_uuid: "ac522642c461494b8c6d5773380a5005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } } }
I20260619 06:42:58.714027 22190 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: This master's current role is: LEADER
I20260619 06:42:58.714828 22193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:42:58.724737 22193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:42:58.726568 22193 catalog_manager.cc:1259] Loaded cluster ID: cbecd308bde742c68a14a13c9abfd94c
I20260619 06:42:58.726784 22193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:42:58.730320 22193 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 06:42:58.744728 22205 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:42:58.747828 22193 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 06:42:58.755128 22193 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:42:58.756477 22193 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Loaded TSK: 0
I20260619 06:42:58.757683 22193 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:42:59.007450 22186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:59.007771 22186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:59.007915 22186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:59.015421 22186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:59.015563 22186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:59.015784 22186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:42:59.025911 22186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:44551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=38825
--tserver_master_addrs=127.7.146.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:59.027273 22186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:59.028625 22186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:59.044468 22212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:59.044656 22214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:59.044459 22211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:59.046365 22186 server_base.cc:1034] running on GCE node
I20260619 06:42:59.047008 22186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:59.048033 22186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:59.049237 22186 hybrid_clock.cc:648] HybridClock initialized: now 1781851379049165 us; error 67 us; skew 500 ppm
I20260619 06:42:59.049542 22186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:59 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:59.052745 22186 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:42:59.054222 22186 webserver.cc:460] Webserver started at http://127.7.146.193:38825/ using document root <none> and password file <none>
I20260619 06:42:59.054903 22186 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:59.055028 22186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:59.060488 22186 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:42:59.063333 22221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:59.064805 22186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:42:59.064957 22186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "2c00f0819a994c7fa1d68795889d3d7e"
format_stamp: "Formatted at 2026-06-19 06:42:56 on dist-test-slave-qfk4"
server_key: "189a14aabbf1bddcdf343457d2228cb1"
server_key_iv: "2b23276842f8235847809b56b50169c6"
server_key_version: "encryptionkey@0"
I20260619 06:42:59.065907 22186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:59.082549 22186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:59.086730 22186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:59.086989 22186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:59.087968 22186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:59.089445 22186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:59.089550 22186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:59.089664 22186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:59.089735 22186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:59.125244 22186 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:44551
I20260619 06:42:59.125325 22333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:44551 every 8 connection(s)
I20260619 06:42:59.127103 22186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:42:59.131367 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 22186
I20260619 06:42:59.132002 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:40867
--local_ip_for_outbound_sockets=127.7.146.194
--tserver_master_addrs=127.7.146.254:40389
--webserver_port=35447
--webserver_interface=127.7.146.194
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:42:59 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:59.157004 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:59.130954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:32787)
0619 06:42:59.131763 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:59.131812 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:59.134136 (+ 2324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:59.135789 (+ 1653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:59.135797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:59.136303 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:59.136980 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:59.137004 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:59.139292 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:59.139299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:59.141277 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:59.141294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:59.143020 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:59.143028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:59.143318 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:59.144039 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:59.144068 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:59.146878 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:59.151420 (+ 4542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:59.151431 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:59.151450 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:59.152077 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:59.152705 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:59.152710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:59.152714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:59.152980 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:59.154103 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:59.154112 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:59.154580 (+ 468us) client_negotiation.cc:769] Sending connection context
0619 06:42:59.154886 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 06:42:59.155466 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":812,"threads_started":1}
I20260619 06:42:59.159200 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:59.131678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:32787 (local address 127.7.146.254:40389)
0619 06:42:59.132459 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:59.132474 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:59.132525 (+ 51us) server_negotiation.cc:407] Connection header received
0619 06:42:59.134762 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:59.134782 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:59.135253 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:59.135685 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:59.137240 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:59.139065 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:59.141459 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:59.142464 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:59.142646 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:59.147254 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:59.147266 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:59.147297 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:59.147343 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:59.151228 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:59.152270 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:59.152280 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:59.152288 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:59.152524 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:59.153129 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:59.153147 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:59.153150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:59.153828 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:59.154023 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:59.157534 (+ 3511us) server_negotiation.cc:299] Negotiation successful
0619 06:42:59.157983 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":1022,"threads_started":1}
I20260619 06:42:59.163277 22334 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40389
I20260619 06:42:59.163650 22334 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:59.164592 22334 heartbeater.cc:511] Master 127.7.146.254:40389 requested a full tablet report, sending...
I20260619 06:42:59.166826 22149 ts_manager.cc:194] Registered new tserver with Master: 2c00f0819a994c7fa1d68795889d3d7e (127.7.146.193:44551)
I20260619 06:42:59.168301 22149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:32787
W20260619 06:42:59.451954 22338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:59.452255 22338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:59.452391 22338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:59.459887 22338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:59.460023 22338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:59.460191 22338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:42:59.470150 22338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:40867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=35447
--tserver_master_addrs=127.7.146.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:59.471472 22338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:59.472805 22338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:59.485947 22343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:59.485867 22344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:59.485970 22346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:59.491760 22338 server_base.cc:1034] running on GCE node
I20260619 06:42:59.492439 22338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:59.493474 22338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:59.494699 22338 hybrid_clock.cc:648] HybridClock initialized: now 1781851379494658 us; error 35 us; skew 500 ppm
I20260619 06:42:59.495025 22338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:59 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:59.498108 22338 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:42:59.499709 22338 webserver.cc:460] Webserver started at http://127.7.146.194:35447/ using document root <none> and password file <none>
I20260619 06:42:59.500425 22338 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:59.500553 22338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:59.505977 22338 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:42:59.508986 22353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:59.510301 22338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:42:59.510452 22338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "3c98a2a5f0f4482da42b0cd37347fe1b"
format_stamp: "Formatted at 2026-06-19 06:42:56 on dist-test-slave-qfk4"
server_key: "7c0a8cd576ea2a55ae69f1be975a7613"
server_key_iv: "191447ed0eb06bf52d990253ec7d7773"
server_key_version: "encryptionkey@0"
I20260619 06:42:59.511416 22338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:59.538561 22338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:59.542572 22338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:59.542827 22338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:59.543798 22338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:59.545133 22338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:59.545228 22338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:59.545332 22338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:59.545398 22338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:59.574965 22338 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:40867
I20260619 06:42:59.575017 22465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:40867 every 8 connection(s)
I20260619 06:42:59.576412 22338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:42:59.577039 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 22338
I20260619 06:42:59.577773 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:42377
--local_ip_for_outbound_sockets=127.7.146.195
--tserver_master_addrs=127.7.146.254:40389
--webserver_port=40245
--webserver_interface=127.7.146.195
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:42:59 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:42:59.596311 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:59.579199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:49675 (local address 127.7.146.254:40389)
0619 06:42:59.579369 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 06:42:59.579374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:42:59.580878 (+ 1504us) server_negotiation.cc:407] Connection header received
0619 06:42:59.582748 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:42:59.582754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:42:59.582838 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:42:59.582927 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:42:59.584673 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:59.585374 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:59.586947 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:42:59.587374 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:42:59.587482 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:59.590846 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:42:59.590851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:42:59.590854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:42:59.590889 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:42:59.593385 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:59.594078 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:59.594084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:59.594087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:59.594219 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:42:59.594626 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:42:59.594630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:42:59.594632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:42:59.594896 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:42:59.595038 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:42:59.595730 (+ 692us) server_negotiation.cc:299] Negotiation successful
0619 06:42:59.595958 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:42:59.597273 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:42:59.579820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:49675)
0619 06:42:59.580697 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 06:42:59.580747 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:42:59.582441 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:42:59.583190 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:42:59.583199 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:42:59.583825 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:42:59.584473 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:59.584499 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:59.585534 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:59.585538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:59.586783 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:42:59.586795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:42:59.587597 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:42:59.587604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:42:59.587828 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:42:59.588460 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:42:59.588491 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:42:59.590647 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:42:59.593527 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:59.593536 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:59.593553 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:59.593954 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:59.594321 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:42:59.594326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:42:59.594330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:42:59.594516 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:42:59.595022 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:42:59.595030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:42:59.595360 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 06:42:59.595599 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 06:42:59.596116 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":330,"threads_started":1}
I20260619 06:42:59.598732 22466 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40389
I20260619 06:42:59.599014 22466 heartbeater.cc:464] Registering TS with master...
I20260619 06:42:59.599690 22466 heartbeater.cc:511] Master 127.7.146.254:40389 requested a full tablet report, sending...
I20260619 06:42:59.601172 22149 ts_manager.cc:194] Registered new tserver with Master: 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867)
I20260619 06:42:59.602321 22149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:49675
W20260619 06:42:59.896390 22469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:42:59.896706 22469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:42:59.896811 22469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:42:59.903949 22469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:42:59.904052 22469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:42:59.904191 22469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:42:59.913923 22469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:42377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=40245
--tserver_master_addrs=127.7.146.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:42:59.915297 22469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:42:59.916672 22469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:42:59.929137 22475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:59.929075 22474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:42:59.929100 22477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:42:59.929867 22469 server_base.cc:1034] running on GCE node
I20260619 06:42:59.930467 22469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:42:59.931476 22469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:42:59.932657 22469 hybrid_clock.cc:648] HybridClock initialized: now 1781851379932607 us; error 44 us; skew 500 ppm
I20260619 06:42:59.932950 22469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:42:59 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:42:59.935928 22469 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:42:59.937466 22469 webserver.cc:460] Webserver started at http://127.7.146.195:40245/ using document root <none> and password file <none>
I20260619 06:42:59.938130 22469 fs_manager.cc:359] Metadata directory not provided
I20260619 06:42:59.938268 22469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:42:59.943641 22469 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:42:59.946363 22484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:59.947780 22469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:42:59.947929 22469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "2ff3edebbbb145d89b15575d6c8ee0dd"
format_stamp: "Formatted at 2026-06-19 06:42:57 on dist-test-slave-qfk4"
server_key: "8c6c0c6273686c8f7d4e5e35e4723efc"
server_key_iv: "e66aabd6114503ad6ef706c5448ae458"
server_key_version: "encryptionkey@0"
I20260619 06:42:59.948941 22469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:42:59.961707 22469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:42:59.965806 22469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:42:59.966046 22469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:42:59.966993 22469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:42:59.968309 22469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:42:59.968408 22469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:59.968508 22469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:42:59.968601 22469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:42:59.997714 22469 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:42377
I20260619 06:42:59.997771 22596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:42377 every 8 connection(s)
I20260619 06:42:59.999102 22469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:43:00.009017 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 22469
Jun 19 06:43:00 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851379, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:43:00.022121 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.001651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:34353 (local address 127.7.146.254:40389)
0619 06:43:00.001811 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.001815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.002908 (+ 1093us) server_negotiation.cc:407] Connection header received
0619 06:43:00.004737 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.004743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.004825 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.004912 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:00.006567 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.007392 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.009642 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.010223 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.010373 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.014914 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:00.014920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:00.014923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:00.014963 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:00.018495 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:00.019314 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:00.019319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:00.019321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:00.019455 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:00.020042 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:00.020047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:00.020049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:00.020404 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:00.020557 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.021608 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.021872 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:43:00.023521 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.002057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:34353)
0619 06:43:00.002745 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.002783 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:00.004447 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:00.005169 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:00.005177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:00.005714 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:43:00.006339 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.006359 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.007575 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.007579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.009410 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.009433 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.010861 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.010870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.011257 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.011953 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:43:00.011979 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:43:00.014666 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:43:00.018662 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:00.018673 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:00.018690 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:00.019177 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:00.019571 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:00.019576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:00.019580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:00.019861 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:00.020540 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:43:00.020546 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:43:00.021087 (+ 541us) client_negotiation.cc:769] Sending connection context
0619 06:43:00.021423 (+ 336us) client_negotiation.cc:240] Negotiation successful
0619 06:43:00.022021 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":249,"threads_started":1}
I20260619 06:43:00.025301 22597 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40389
I20260619 06:43:00.025682 22597 heartbeater.cc:464] Registering TS with master...
I20260619 06:43:00.026473 22597 heartbeater.cc:511] Master 127.7.146.254:40389 requested a full tablet report, sending...
I20260619 06:43:00.026567 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.010126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.7.146.254:40389)
0619 06:43:00.011320 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.011325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.011340 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:43:00.011385 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.011389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.011454 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.011552 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:00.013219 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.014246 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.016061 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.016940 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.017085 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.019747 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:00.019753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:00.019757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:00.019794 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:00.023288 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:00.024354 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:00.024359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:00.024362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:00.024505 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:00.024961 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:00.024966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:00.024968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:00.025315 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:00.025647 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.025936 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.026194 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1094,"thread_start_us":942,"threads_started":1}
I20260619 06:43:00.028129 22149 ts_manager.cc:194] Registered new tserver with Master: 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377)
I20260619 06:43:00.029122 22149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:34353
I20260619 06:43:00.029510 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:43:00.036540 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.030585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.7.146.254:40389)
0619 06:43:00.030756 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.030761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.031249 (+ 488us) server_negotiation.cc:407] Connection header received
0619 06:43:00.031481 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.031485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.031617 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.031748 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:43:00.032196 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.032947 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.034371 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.034793 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.034904 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.035431 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:43:00.035947 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:43:00.036087 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.036199 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.036289 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:43:00.037592 22149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:43:00.044997 22149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:43:00.073551 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.060326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.7.146.194:40867)
0619 06:43:00.065748 (+ 5422us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.065758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.065796 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:43:00.065910 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.065919 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.066237 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.066477 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:00.067157 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.068746 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.071112 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.072444 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.072594 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.072826 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.073048 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.073225 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5229,"thread_start_us":5081,"threads_started":1}
I20260619 06:43:00.073668 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.059979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:40867 (local address 127.0.0.1:49176)
0619 06:43:00.065057 (+ 5078us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.065078 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:00.065203 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:00.066513 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:00.066521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:00.066551 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:00.066939 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.066952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.068885 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.068890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.070969 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.070980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.072650 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.072656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.072832 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.072881 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 06:43:00.072990 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 06:43:00.073181 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4932,"thread_start_us":321,"threads_started":1}
I20260619 06:43:00.074409 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.058308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.0.0.1:55074)
0619 06:43:00.060634 (+ 2326us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.060660 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:00.060783 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:00.066976 (+ 6193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:00.066981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:00.067013 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:00.067488 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.067500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.069224 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.069230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.071706 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.071715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.073675 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.073683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.073899 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.073929 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:43:00.074009 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:43:00.074133 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2141,"mutex_wait_us":116,"thread_start_us":708,"threads_started":1}
I20260619 06:43:00.075165 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.058078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:42377 (local address 127.0.0.1:35146)
0619 06:43:00.059041 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.059094 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:00.059292 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:00.066773 (+ 7481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:00.066778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:00.066802 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:00.067281 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.067295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.069278 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.069282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.071701 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.071715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.073696 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.073701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.074594 (+ 893us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.074624 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:43:00.074700 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:43:00.074823 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":753,"thread_start_us":6661,"threads_started":1}
I20260619 06:43:00.075233 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.058782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.7.146.195:42377)
0619 06:43:00.066141 (+ 7359us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.066150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.066184 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:43:00.066282 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.066289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.066547 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.067480 (+ 933us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:00.067750 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.069118 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.071878 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.073303 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.073504 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.074070 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.074758 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.074921 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7190,"thread_start_us":7033,"threads_started":1}
I20260619 06:43:00.076177 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.058598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.7.146.193:44551)
0619 06:43:00.066414 (+ 7816us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.066430 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.066454 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:43:00.066544 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.066550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.066775 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.066965 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:00.067697 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.069104 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.071879 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.073303 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.075479 (+ 2176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.075667 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.075726 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.075878 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7674,"thread_start_us":7545,"threads_started":1}
I20260619 06:43:00.077648 22401 tablet_service.cc:1467] Processing CreateTablet for tablet 9f189f5a495447578703602e0f1ac347 (DEFAULT_TABLE table=test-table [id=6dd2f2a58fd244519f3069b6cebe7738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:43:00.079180 22401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f189f5a495447578703602e0f1ac347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:43:00.079072 22269 tablet_service.cc:1467] Processing CreateTablet for tablet 9f189f5a495447578703602e0f1ac347 (DEFAULT_TABLE table=test-table [id=6dd2f2a58fd244519f3069b6cebe7738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:43:00.078945 22532 tablet_service.cc:1467] Processing CreateTablet for tablet 9f189f5a495447578703602e0f1ac347 (DEFAULT_TABLE table=test-table [id=6dd2f2a58fd244519f3069b6cebe7738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:43:00.080042 22269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f189f5a495447578703602e0f1ac347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:43:00.080264 22532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f189f5a495447578703602e0f1ac347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:43:00.092010 22613 tablet_bootstrap.cc:492] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Bootstrap starting.
I20260619 06:43:00.094357 22614 tablet_bootstrap.cc:492] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: Bootstrap starting.
I20260619 06:43:00.095644 22613 tablet_bootstrap.cc:654] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Neither blocks nor log segments found. Creating new log.
I20260619 06:43:00.095876 22615 tablet_bootstrap.cc:492] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: Bootstrap starting.
I20260619 06:43:00.096966 22613 log.cc:824] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:00.097811 22614 tablet_bootstrap.cc:654] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: Neither blocks nor log segments found. Creating new log.
I20260619 06:43:00.098935 22615 tablet_bootstrap.cc:654] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: Neither blocks nor log segments found. Creating new log.
I20260619 06:43:00.099049 22614 log.cc:824] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:00.099987 22615 log.cc:824] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:00.101012 22613 tablet_bootstrap.cc:492] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: No bootstrap required, opened a new log
I20260619 06:43:00.101255 22613 ts_tablet_manager.cc:1397] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 06:43:00.103025 22615 tablet_bootstrap.cc:492] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: No bootstrap required, opened a new log
I20260619 06:43:00.103259 22615 ts_tablet_manager.cc:1397] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260619 06:43:00.106076 22614 tablet_bootstrap.cc:492] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: No bootstrap required, opened a new log
I20260619 06:43:00.106348 22614 ts_tablet_manager.cc:1397] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260619 06:43:00.111666 22613 raft_consensus.cc:348] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:00.112167 22613 raft_consensus.cc:374] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:43:00.112264 22613 raft_consensus.cc:729] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c00f0819a994c7fa1d68795889d3d7e, State: Initialized, Role: FOLLOWER
I20260619 06:43:00.113078 22613 consensus_queue.cc:260] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:00.113385 22615 raft_consensus.cc:348] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:00.113816 22615 raft_consensus.cc:374] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:43:00.113915 22615 raft_consensus.cc:729] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff3edebbbb145d89b15575d6c8ee0dd, State: Initialized, Role: FOLLOWER
I20260619 06:43:00.113878 22614 raft_consensus.cc:348] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:00.114439 22614 raft_consensus.cc:374] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:43:00.114436 22334 heartbeater.cc:503] Master 127.7.146.254:40389 was elected leader, sending a full tablet report...
I20260619 06:43:00.114532 22614 raft_consensus.cc:729] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c98a2a5f0f4482da42b0cd37347fe1b, State: Initialized, Role: FOLLOWER
I20260619 06:43:00.114508 22615 consensus_queue.cc:260] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:00.115036 22613 ts_tablet_manager.cc:1428] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 06:43:00.115223 22614 consensus_queue.cc:260] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:00.115979 22597 heartbeater.cc:503] Master 127.7.146.254:40389 was elected leader, sending a full tablet report...
I20260619 06:43:00.116546 22615 ts_tablet_manager.cc:1428] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260619 06:43:00.118000 22466 heartbeater.cc:503] Master 127.7.146.254:40389 was elected leader, sending a full tablet report...
I20260619 06:43:00.118621 22614 ts_tablet_manager.cc:1428] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
W20260619 06:43:00.129724 22335 tablet.cc:2369] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:43:00.250707 22598 tablet.cc:2369] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:43:00.265763 22621 raft_consensus.cc:482] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:43:00.266060 22621 raft_consensus.cc:504] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:00.267693 22621 leader_election.cc:288] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377), 2c00f0819a994c7fa1d68795889d3d7e (127.7.146.193:44551)
I20260619 06:43:00.274369 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.268175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:55083 (local address 127.7.146.193:44551)
0619 06:43:00.268366 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.268372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.269162 (+ 790us) server_negotiation.cc:407] Connection header received
0619 06:43:00.269349 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.269353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.269419 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.269494 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:00.270081 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.270922 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.272751 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.273494 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.273667 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.273743 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.273996 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.274122 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:43:00.274467 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.268137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:53383 (local address 127.7.146.195:42377)
0619 06:43:00.268340 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.268345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.269268 (+ 923us) server_negotiation.cc:407] Connection header received
0619 06:43:00.269493 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.269497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.269554 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.269629 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:00.270261 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.271019 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.272792 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.273563 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.273713 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.273785 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.274085 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.274222 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:43:00.274528 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.268103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.7.146.194:55083)
0619 06:43:00.269049 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.269070 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:00.269215 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:00.269534 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:00.269539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:00.269562 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:00.269915 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.269925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.271052 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.271057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.272608 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.272622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.273691 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.273697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.273845 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.273869 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:43:00.273951 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:43:00.274090 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":3,"spinlock_wait_cycles":18176,"thread_start_us":565,"threads_started":1}
I20260619 06:43:00.274562 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.268247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:42377 (local address 127.7.146.194:53383)
0619 06:43:00.269162 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.269177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:00.269360 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:00.269667 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:00.269671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:00.269709 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:00.270096 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.270107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.271150 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.271155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.272644 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:00.272660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.273769 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.273776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:00.273941 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.273963 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:43:00.274039 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:43:00.274154 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":21,"thread_start_us":558,"threads_started":1}
I20260619 06:43:00.275583 22552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f189f5a495447578703602e0f1ac347" candidate_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" is_pre_election: true
I20260619 06:43:00.275583 22289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f189f5a495447578703602e0f1ac347" candidate_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c00f0819a994c7fa1d68795889d3d7e" is_pre_election: true
I20260619 06:43:00.276160 22289 raft_consensus.cc:2454] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c98a2a5f0f4482da42b0cd37347fe1b in term 0.
I20260619 06:43:00.276170 22552 raft_consensus.cc:2454] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c98a2a5f0f4482da42b0cd37347fe1b in term 0.
I20260619 06:43:00.276909 22357 leader_election.cc:302] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff3edebbbb145d89b15575d6c8ee0dd, 3c98a2a5f0f4482da42b0cd37347fe1b; no voters:
I20260619 06:43:00.277304 22621 raft_consensus.cc:2781] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:43:00.277451 22621 raft_consensus.cc:482] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:43:00.277561 22621 raft_consensus.cc:3037] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:43:00.279783 22621 raft_consensus.cc:504] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:00.280449 22621 leader_election.cc:288] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [CANDIDATE]: Term 1 election: Requested vote from peers 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377), 2c00f0819a994c7fa1d68795889d3d7e (127.7.146.193:44551)
I20260619 06:43:00.281025 22289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f189f5a495447578703602e0f1ac347" candidate_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c00f0819a994c7fa1d68795889d3d7e"
I20260619 06:43:00.281032 22552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f189f5a495447578703602e0f1ac347" candidate_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd"
I20260619 06:43:00.281255 22552 raft_consensus.cc:3037] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:43:00.281255 22289 raft_consensus.cc:3037] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:43:00.284399 22552 raft_consensus.cc:2454] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c98a2a5f0f4482da42b0cd37347fe1b in term 1.
I20260619 06:43:00.284399 22289 raft_consensus.cc:2454] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c98a2a5f0f4482da42b0cd37347fe1b in term 1.
I20260619 06:43:00.284895 22357 leader_election.cc:302] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c00f0819a994c7fa1d68795889d3d7e, 2ff3edebbbb145d89b15575d6c8ee0dd, 3c98a2a5f0f4482da42b0cd37347fe1b; no voters:
I20260619 06:43:00.285248 22621 raft_consensus.cc:2781] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:43:00.285725 22621 raft_consensus.cc:686] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 1 LEADER]: Becoming Leader. State: Replica: 3c98a2a5f0f4482da42b0cd37347fe1b, State: Running, Role: LEADER
I20260619 06:43:00.286181 22621 consensus_queue.cc:237] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:00.290712 22148 catalog_manager.cc:5495] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194). New cstate: current_term: 1 leader_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } health_report { overall_health: HEALTHY } } }
I20260619 06:43:00.309662 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.304801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.7.146.194:40867)
0619 06:43:00.304979 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.305006 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.305024 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:43:00.305112 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.305115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.305190 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.305317 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:43:00.305615 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.306331 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.307688 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.308150 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.308244 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.308704 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:43:00.309214 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:43:00.309318 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.309394 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.309460 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:43:00.318372 22289 raft_consensus.cc:1261] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Refusing update from remote peer 3c98a2a5f0f4482da42b0cd37347fe1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:43:00.318372 22552 raft_consensus.cc:1261] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 1 FOLLOWER]: Refusing update from remote peer 3c98a2a5f0f4482da42b0cd37347fe1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:43:00.319491 22624 consensus_queue.cc:1035] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:43:00.319905 22621 consensus_queue.cc:1035] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:43:00.328413 22467 tablet.cc:2369] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:43:00.344281 22149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:43:00.344533 22149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:43:00.348057 22149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Sending DeleteTablet for 3 replicas of tablet 9f189f5a495447578703602e0f1ac347
I20260619 06:43:00.349236 22269 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f189f5a495447578703602e0f1ac347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:43:00 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:55074
I20260619 06:43:00.349241 22401 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f189f5a495447578703602e0f1ac347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:43:00 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:49176
I20260619 06:43:00.349251 22532 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f189f5a495447578703602e0f1ac347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:43:00 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:35146
I20260619 06:43:00.352393 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 22186
I20260619 06:43:00.366861 22637 tablet_replica.cc:330] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: stopping tablet replica
I20260619 06:43:00.367411 22637 raft_consensus.cc:2229] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:43:00.367987 22637 raft_consensus.cc:2258] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:43:00.370661 22637 ts_tablet_manager.cc:1905] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:43:00.371511 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.353554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.7.146.193:44551)
0619 06:43:00.353747 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.353753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.353775 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:43:00.353837 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.353842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.353930 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.354045 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:00.355871 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.357082 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.359233 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.359918 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.360059 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.363864 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:00.363885 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:00.363905 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:00.363954 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:00.367948 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:00.368929 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:00.368939 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:00.368943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:00.369113 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:00.369563 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:00.369569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:00.369572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:00.370092 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:00.370399 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.370941 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.371175 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:43:00.373371 22329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:00.374851 22636 tablet_replica.cc:330] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: stopping tablet replica
I20260619 06:43:00.376005 22636 raft_consensus.cc:2229] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:43:00.377359 22636 raft_consensus.cc:2258] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:43:00.378173 22637 ts_tablet_manager.cc:1918] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:43:00.378340 22637 log.cc:1196] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/9f189f5a495447578703602e0f1ac347
I20260619 06:43:00.378893 22637 ts_tablet_manager.cc:1939] T 9f189f5a495447578703602e0f1ac347 P 2ff3edebbbb145d89b15575d6c8ee0dd: Deleting consensus metadata
I20260619 06:43:00.381532 22136 catalog_manager.cc:4841] TS 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377): tablet 9f189f5a495447578703602e0f1ac347 (table test-table [id=6dd2f2a58fd244519f3069b6cebe7738]) successfully deleted
I20260619 06:43:00.383601 22636 ts_tablet_manager.cc:1905] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:43:00.394368 22636 ts_tablet_manager.cc:1918] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:43:00.394569 22636 log.cc:1196] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/9f189f5a495447578703602e0f1ac347
I20260619 06:43:00.395151 22636 ts_tablet_manager.cc:1939] T 9f189f5a495447578703602e0f1ac347 P 3c98a2a5f0f4482da42b0cd37347fe1b: Deleting consensus metadata
I20260619 06:43:00.397217 22134 catalog_manager.cc:4841] TS 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867): tablet 9f189f5a495447578703602e0f1ac347 (table test-table [id=6dd2f2a58fd244519f3069b6cebe7738]) successfully deleted
I20260619 06:43:00.482440 22635 tablet_replica.cc:330] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: stopping tablet replica
I20260619 06:43:00.482983 22635 raft_consensus.cc:2229] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:43:00.483279 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 22186
I20260619 06:43:00.499404 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 22338
W20260619 06:43:00.499440 22133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.146.193:44551 (error 108)
I20260619 06:43:00.501382 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.500927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.0.0.1:55096)
0619 06:43:00.501090 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.501201 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:43:00.501650 22133 catalog_manager.cc:4584] TS 2c00f0819a994c7fa1d68795889d3d7e (127.7.146.193:44551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f189f5a495447578703602e0f1ac347: Network error: Client connection negotiation failed: client connection to 127.7.146.193:44551: connect: Connection refused (error 111)
I20260619 06:43:00.516098 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.500709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.7.146.194:40867)
0619 06:43:00.500913 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.500919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.501116 (+ 197us) server_negotiation.cc:407] Connection header received
0619 06:43:00.501205 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.501211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.501307 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.501417 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:00.503162 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.503966 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.505839 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.506321 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.506429 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.509964 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:00.509981 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:00.509998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:00.510040 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:00.513160 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:00.514019 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:00.514026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:00.514029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:00.514194 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:00.514611 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:00.514615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:00.514617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:00.515011 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:00.515273 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.515612 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.515787 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:43:00.516892 22461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:00.555256 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.554732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.0.0.1:55106)
0619 06:43:00.554961 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.555088 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 06:43:00.556273 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.555949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.0.0.1:55114)
0619 06:43:00.556084 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.556166 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 06:43:00.617169 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.616607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.0.0.1:55128)
0619 06:43:00.616830 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.616943 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:43:00.618211 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.617803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.0.0.1:55144)
0619 06:43:00.617961 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.618050 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 06:43:00.633144 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 22338
I20260619 06:43:00.648104 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 22469
I20260619 06:43:00.663676 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.648876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.7.146.195:42377)
0619 06:43:00.649080 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:00.649085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:00.649104 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:43:00.649212 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:00.649216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:00.649279 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:00.649362 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:00.650654 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.651399 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.652711 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:00.653174 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:00.653265 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:00.658540 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:00.658555 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:00.658573 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:00.658617 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:00.661378 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:00.662015 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:00.662023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:00.662025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:00.662189 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:00.662451 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:00.662454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:00.662456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:00.662791 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:00.663000 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:00.663208 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 06:43:00.663366 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:43:00.664271 22592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:00.699010 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.698371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.0.0.1:55150)
0619 06:43:00.698673 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.698791 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 06:43:00.700116 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.699688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.0.0.1:55160)
0619 06:43:00.699898 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.699983 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 06:43:00.753048 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 22469
I20260619 06:43:00.765967 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 22116
I20260619 06:43:00.767220 22179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:00.881134 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 22116
I20260619 06:43:00.881210 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:00.880656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44551 (local address 127.0.0.1:33806)
0619 06:43:00.880876 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:00.880997 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:43:00.898409 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40389
--webserver_interface=127.7.146.254
--webserver_port=41481
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:40389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:43:01.217693 22643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:01.218011 22643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:01.218144 22643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:01.225584 22643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:43:01.225728 22643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:01.225807 22643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:01.225880 22643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:43:01.225939 22643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:43:01.236207 22643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:40389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:01.237624 22643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:01.238973 22643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:01.250653 22648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:01.250497 22649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:01.250648 22651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:01.251194 22643 server_base.cc:1034] running on GCE node
I20260619 06:43:01.251997 22643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:01.253599 22643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:01.254833 22643 hybrid_clock.cc:648] HybridClock initialized: now 1781851381254761 us; error 63 us; skew 500 ppm
I20260619 06:43:01.255234 22643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:01 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851381, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:01.258380 22643 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:43:01.259934 22643 webserver.cc:460] Webserver started at http://127.7.146.254:41481/ using document root <none> and password file <none>
I20260619 06:43:01.260676 22643 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:01.260797 22643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:01.266180 22643 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:43:01.268998 22658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:01.270355 22643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:43:01.270509 22643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "ac522642c461494b8c6d5773380a5005"
format_stamp: "Formatted at 2026-06-19 06:42:55 on dist-test-slave-qfk4"
server_key: "9f0cba17d7c6c0ccf1dd25c835a9141e"
server_key_iv: "691cada2a2375e6b05f89f34cfe342fc"
server_key_version: "encryptionkey@0"
I20260619 06:43:01.271466 22643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:01.285105 22643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:01.289451 22643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:01.289731 22643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:01.307111 22643 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:40389
I20260619 06:43:01.307130 22709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:40389 every 8 connection(s)
I20260619 06:43:01.308578 22643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:43:01.309399 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 22643
I20260619 06:43:01.310257 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:44551
--local_ip_for_outbound_sockets=127.7.146.193
--tserver_master_addrs=127.7.146.254:40389
--webserver_port=38825
--webserver_interface=127.7.146.193
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:43:01.315470 22710 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:43:01.318809 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Bootstrap starting.
I20260619 06:43:01.339299 22710 log.cc:824] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:01.356649 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:43:01.357342 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005: Bootstrap complete.
I20260619 06:43:01.368160 22710 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:43:01.369192 22710 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac522642c461494b8c6d5773380a5005, State: Initialized, Role: FOLLOWER
I20260619 06:43:01.369853 22710 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:43:01.370107 22710 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:43:01.370216 22710 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:43:01.370383 22710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:43:01.374402 22710 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:43:01.374979 22710 leader_election.cc:302] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac522642c461494b8c6d5773380a5005; no voters:
I20260619 06:43:01.375592 22710 leader_election.cc:288] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 06:43:01.375757 22714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:43:01.377884 22710 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:43:01.381026 22714 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [term 3 LEADER]: Becoming Leader. State: Replica: ac522642c461494b8c6d5773380a5005, State: Running, Role: LEADER
I20260619 06:43:01.381928 22714 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } }
I20260619 06:43:01.384796 22715 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ac522642c461494b8c6d5773380a5005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } } }
I20260619 06:43:01.384802 22716 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac522642c461494b8c6d5773380a5005. Latest consensus state: current_term: 3 leader_uuid: "ac522642c461494b8c6d5773380a5005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac522642c461494b8c6d5773380a5005" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40389 } } }
I20260619 06:43:01.385197 22715 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: This master's current role is: LEADER
I20260619 06:43:01.385203 22716 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac522642c461494b8c6d5773380a5005 [sys.catalog]: This master's current role is: LEADER
I20260619 06:43:01.394014 22726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:43:01.400735 22726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:43:01.401914 22726 catalog_manager.cc:1259] Loaded cluster ID: cbecd308bde742c68a14a13c9abfd94c
I20260619 06:43:01.402014 22726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 06:43:01.403481 22726 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f66219ed9ad google::LogMessage::Fail() at ??:0
@ 0x7f66219f088a google::LogMessage::SendToLog() at ??:0
@ 0x7f66219ed4de google::LogMessage::Flush() at ??:0
@ 0x7f66219eea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6637420512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f663741d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f66374a5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f66374a5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6624639345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f662463c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f662463c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6624617eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f66278f16db start_thread at ??:0
@ 0x7f662088761f clone at ??:0
W20260619 06:43:01.704133 22712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:01.704464 22712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:01.704630 22712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:01.712182 22712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:01.712316 22712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:01.712450 22712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:43:01.722862 22712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:44551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=38825
--tserver_master_addrs=127.7.146.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:01.724257 22712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:01.725643 22712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:01.740366 22751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:01.740406 22752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:01.740443 22754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:01.741257 22712 server_base.cc:1034] running on GCE node
I20260619 06:43:01.741875 22712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:01.742822 22712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:01.744134 22712 hybrid_clock.cc:648] HybridClock initialized: now 1781851381744074 us; error 49 us; skew 500 ppm
I20260619 06:43:01.744529 22712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:01 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851381, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:01.747772 22712 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:43:01.749413 22712 webserver.cc:460] Webserver started at http://127.7.146.193:38825/ using document root <none> and password file <none>
I20260619 06:43:01.750120 22712 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:01.750236 22712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:01.755970 22712 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:43:01.759255 22761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:01.760794 22712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:43:01.760957 22712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "2c00f0819a994c7fa1d68795889d3d7e"
format_stamp: "Formatted at 2026-06-19 06:42:56 on dist-test-slave-qfk4"
server_key: "189a14aabbf1bddcdf343457d2228cb1"
server_key_iv: "2b23276842f8235847809b56b50169c6"
server_key_version: "encryptionkey@0"
I20260619 06:43:01.762001 22712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:01.785689 22712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:01.789920 22712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:01.790171 22712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:01.791186 22712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:43:01.793025 22768 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 06:43:01.798520 22712 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 06:43:01.798633 22712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 06:43:01.798779 22712 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 06:43:01.801987 22712 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 06:43:01.802071 22712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260619 06:43:01.802399 22768 tablet_bootstrap.cc:492] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Bootstrap starting.
I20260619 06:43:01.828625 22768 log.cc:824] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:01.838095 22712 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:44551
I20260619 06:43:01.838130 22875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:44551 every 8 connection(s)
I20260619 06:43:01.839447 22712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:43:01.840121 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 22712
I20260619 06:43:01.840936 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:40867
--local_ip_for_outbound_sockets=127.7.146.194
--tserver_master_addrs=127.7.146.254:40389
--webserver_port=35447
--webserver_interface=127.7.146.194
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:43:01.842097 22768 tablet_bootstrap.cc:492] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:43:01.842582 22768 tablet_bootstrap.cc:492] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Bootstrap complete.
I20260619 06:43:01.843642 22768 ts_tablet_manager.cc:1397] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Time spent bootstrapping tablet: real 0.042s user 0.025s sys 0.012s
I20260619 06:43:01.848733 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:01.846158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:52525)
0619 06:43:01.847149 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:01.847476 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":456,"threads_started":1}
W20260619 06:43:01.849434 22876 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:40389 (0 consecutive failures): Network error: Failed to ping master at 127.7.146.254:40389: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
I20260619 06:43:01.850414 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:01.850025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:52543)
0619 06:43:01.850180 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:01.850279 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 06:43:01.851460 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:01.851076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:49369)
0619 06:43:01.851245 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:01.851328 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 06:43:01.851724 22876 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:43:01.852495 22768 raft_consensus.cc:348] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:01.853677 22768 raft_consensus.cc:729] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c00f0819a994c7fa1d68795889d3d7e, State: Initialized, Role: FOLLOWER
I20260619 06:43:01.854307 22768 consensus_queue.cc:260] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:01.855731 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:01.855407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:47071)
0619 06:43:01.855543 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:01.855611 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 06:43:01.855865 22768 ts_tablet_manager.cc:1428] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260619 06:43:01.856961 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:01.856550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:48745)
0619 06:43:01.856743 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:01.856820 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260619 06:43:02.164989 22879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:02.165392 22879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:02.165566 22879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:02.175995 22879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:02.176121 22879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:02.176239 22879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:43:02.187551 22879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:40867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=35447
--tserver_master_addrs=127.7.146.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:02.188925 22879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:02.190255 22879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:02.203076 22885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:02.203055 22886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:02.203116 22888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:02.203951 22879 server_base.cc:1034] running on GCE node
I20260619 06:43:02.204627 22879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:02.205647 22879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:02.206849 22879 hybrid_clock.cc:648] HybridClock initialized: now 1781851382206792 us; error 52 us; skew 500 ppm
I20260619 06:43:02.207168 22879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:02 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851382, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:02.210498 22879 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:43:02.211953 22879 webserver.cc:460] Webserver started at http://127.7.146.194:35447/ using document root <none> and password file <none>
I20260619 06:43:02.212725 22879 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:02.212850 22879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:02.218362 22879 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:43:02.221349 22895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:02.222780 22879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:43:02.222934 22879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "3c98a2a5f0f4482da42b0cd37347fe1b"
format_stamp: "Formatted at 2026-06-19 06:42:56 on dist-test-slave-qfk4"
server_key: "7c0a8cd576ea2a55ae69f1be975a7613"
server_key_iv: "191447ed0eb06bf52d990253ec7d7773"
server_key_version: "encryptionkey@0"
I20260619 06:43:02.223942 22879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:02.241417 22879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:02.245534 22879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:02.245782 22879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:02.246824 22879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:43:02.248137 22879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:43:02.248232 22879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:02.248325 22879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:43:02.248391 22879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:02.278417 22879 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:40867
I20260619 06:43:02.278561 23007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:40867 every 8 connection(s)
I20260619 06:43:02.279847 22879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:43:02.284770 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:02.282779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:38625)
0619 06:43:02.283566 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:02.283817 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":284,"threads_started":1}
W20260619 06:43:02.285490 23008 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:40389 (0 consecutive failures): Network error: Failed to ping master at 127.7.146.254:40389: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
I20260619 06:43:02.286317 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:02.285965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:55783)
0619 06:43:02.286124 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:02.286206 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:43:02.287168 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:02.286807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:54999)
0619 06:43:02.286952 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:02.287037 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 06:43:02.287406 23008 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:43:02.287626 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 22879
I20260619 06:43:02.288206 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:42377
--local_ip_for_outbound_sockets=127.7.146.195
--tserver_master_addrs=127.7.146.254:40389
--webserver_port=40245
--webserver_interface=127.7.146.195
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:43:02.601958 23011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:02.602228 23011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:02.602339 23011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:02.609557 23011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:02.609671 23011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:02.609809 23011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:43:02.619598 23011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:42377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=40245
--tserver_master_addrs=127.7.146.254:40389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:02.620925 23011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:02.622249 23011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:02.635113 23019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:02.635121 23017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:02.635121 23016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:02.635941 23011 server_base.cc:1034] running on GCE node
I20260619 06:43:02.636530 23011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:02.637473 23011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:02.638676 23011 hybrid_clock.cc:648] HybridClock initialized: now 1781851382638619 us; error 54 us; skew 500 ppm
I20260619 06:43:02.638970 23011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:02 dist-test-slave-qfk4 krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851382, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:02.641995 23011 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:43:02.643400 23011 webserver.cc:460] Webserver started at http://127.7.146.195:40245/ using document root <none> and password file <none>
I20260619 06:43:02.644075 23011 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:02.644186 23011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:02.649314 23011 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:43:02.651966 23026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:02.653275 23011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:43:02.653424 23011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "2ff3edebbbb145d89b15575d6c8ee0dd"
format_stamp: "Formatted at 2026-06-19 06:42:57 on dist-test-slave-qfk4"
server_key: "8c6c0c6273686c8f7d4e5e35e4723efc"
server_key_iv: "e66aabd6114503ad6ef706c5448ae458"
server_key_version: "encryptionkey@0"
I20260619 06:43:02.654346 23011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:02.673799 23011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:02.677767 23011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:02.678013 23011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:02.678998 23011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:43:02.680308 23011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:43:02.680398 23011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:02.680505 23011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:43:02.680590 23011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:02.711095 23011 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:42377
I20260619 06:43:02.711153 23138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:42377 every 8 connection(s)
I20260619 06:43:02.712494 23011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:43:02.717247 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:02.715331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:50873)
0619 06:43:02.716036 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:02.716283 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":263,"threads_started":1}
W20260619 06:43:02.717881 23139 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:40389 (0 consecutive failures): Network error: Failed to ping master at 127.7.146.254:40389: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
I20260619 06:43:02.718724 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:02.718333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:52697)
0619 06:43:02.718513 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:02.718609 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 06:43:02.719374 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 23011
I20260619 06:43:02.719637 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:02.719228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:42559)
0619 06:43:02.719380 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:02.719484 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:43:02.719866 23139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:43:02.858705 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:02.857813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:49333)
0619 06:43:02.858374 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:02.858499 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":298,"threads_started":1}
I20260619 06:43:03.288843 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:03.287966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:60133)
0619 06:43:03.288431 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:03.288542 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":220,"threads_started":1}
I20260619 06:43:03.296348 23145 raft_consensus.cc:482] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:43:03.296645 23145 raft_consensus.cc:504] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:03.298241 23145 leader_election.cc:288] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377), 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867)
Jun 19 06:43:03 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851381, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
Jun 19 06:43:03 dist-test-slave-qfk4 krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851381, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:43:03.320012 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:03.298484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:40867 (local address 127.7.146.193:34461)
0619 06:43:03.298710 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:03.298754 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:03.299338 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:03.302302 (+ 2964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:03.302312 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:03.303235 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:43:03.303985 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:03.304006 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:03.305567 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:03.305571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:03.306949 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:03.306961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:03.307910 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:03.307917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:03.308131 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:03.308932 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:43:03.308949 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:43:03.311378 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:43:03.315731 (+ 4353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:03.315740 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:03.315756 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:03.316490 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:03.316993 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:03.316999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:03.317002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:03.317298 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:03.318216 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:43:03.318224 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:43:03.318675 (+ 451us) client_negotiation.cc:769] Sending connection context
0619 06:43:03.318994 (+ 319us) client_negotiation.cc:240] Negotiation successful
0619 06:43:03.319553 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":213,"spinlock_wait_cycles":79872}
I20260619 06:43:03.320031 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:03.298810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:34461 (local address 127.7.146.194:40867)
0619 06:43:03.300031 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:03.300042 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:03.300075 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:43:03.301616 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:03.301625 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:03.301946 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:03.302269 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:03.304177 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:03.305385 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:03.307080 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:03.307687 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:03.307846 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:03.311660 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:03.311668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:03.311682 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:03.311717 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:03.315578 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:03.316639 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:03.316647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:03.316654 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:03.316869 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:03.317402 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:03.317407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:03.317411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:03.318067 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:03.318251 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:03.319248 (+ 997us) server_negotiation.cc:299] Negotiation successful
0619 06:43:03.319634 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":897,"threads_started":1}
I20260619 06:43:03.320031 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:03.298740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:45033 (local address 127.7.146.195:42377)
0619 06:43:03.299235 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:03.299252 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:03.299845 (+ 593us) server_negotiation.cc:407] Connection header received
0619 06:43:03.301458 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:03.301467 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:03.301843 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:03.302110 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:03.304180 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:03.305385 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:03.307079 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:03.307675 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:03.307835 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:03.311936 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:03.311945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:03.311962 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:03.312009 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:03.315578 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:03.316638 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:03.316670 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:03.316675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:03.316866 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:03.317402 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:03.317407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:03.317411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:03.318076 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:03.318240 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:03.319222 (+ 982us) server_negotiation.cc:299] Negotiation successful
0619 06:43:03.319625 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":297,"threads_started":1}
I20260619 06:43:03.320108 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:03.298491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:42377 (local address 127.7.146.193:45033)
0619 06:43:03.299716 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:03.299733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:03.299861 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:03.302302 (+ 2441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:03.302311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:03.303243 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:43:03.303984 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:03.304006 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:03.305558 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:03.305562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:03.306951 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:03.306962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:03.307869 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:03.307875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:03.308133 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:03.308883 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:43:03.308911 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:43:03.311699 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:43:03.315734 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:03.315740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:03.315756 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:03.316481 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:03.317005 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:03.317009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:03.317012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:03.317298 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:03.318216 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:43:03.318224 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:43:03.318675 (+ 451us) client_negotiation.cc:769] Sending connection context
0619 06:43:03.318994 (+ 319us) client_negotiation.cc:240] Negotiation successful
0619 06:43:03.319726 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":812,"spinlock_wait_cycles":26112,"thread_start_us":460,"threads_started":1}
W20260619 06:43:03.323421 22765 leader_election.cc:341] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377): Not found: Tablet not found: 9f189f5a495447578703602e0f1ac347
W20260619 06:43:03.323669 22763 leader_election.cc:341] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867): Not found: Tablet not found: 9f189f5a495447578703602e0f1ac347
I20260619 06:43:03.323801 22763 leader_election.cc:302] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c00f0819a994c7fa1d68795889d3d7e; no voters: 2ff3edebbbb145d89b15575d6c8ee0dd, 3c98a2a5f0f4482da42b0cd37347fe1b
I20260619 06:43:03.324167 23145 raft_consensus.cc:2726] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:43:03.721312 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:03.720382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:47731)
0619 06:43:03.720945 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:03.721085 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":357,"threads_started":1}
I20260619 06:43:03.860313 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:03.859591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:58453)
0619 06:43:03.860021 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:03.860111 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":222,"threads_started":1}
I20260619 06:43:04.290416 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:04.289688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:46485)
0619 06:43:04.290169 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:04.290255 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":254,"threads_started":1}
I20260619 06:43:04.722927 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:04.722220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:50603)
0619 06:43:04.722682 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:04.722770 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":239,"threads_started":1}
I20260619 06:43:04.862169 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:04.861266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:50913)
0619 06:43:04.861807 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:04.861925 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":252,"threads_started":1}
I20260619 06:43:05.171653 23154 raft_consensus.cc:482] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:43:05.171860 23154 raft_consensus.cc:504] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:05.172469 23154 leader_election.cc:288] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377), 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867)
W20260619 06:43:05.173616 22763 leader_election.cc:341] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867): Not found: Tablet not found: 9f189f5a495447578703602e0f1ac347
W20260619 06:43:05.173775 22765 leader_election.cc:341] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377): Not found: Tablet not found: 9f189f5a495447578703602e0f1ac347
I20260619 06:43:05.173919 22765 leader_election.cc:302] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c00f0819a994c7fa1d68795889d3d7e; no voters: 2ff3edebbbb145d89b15575d6c8ee0dd, 3c98a2a5f0f4482da42b0cd37347fe1b
I20260619 06:43:05.174149 23154 raft_consensus.cc:2726] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:43:05.291960 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:05.291293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:51087)
0619 06:43:05.291704 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:05.291791 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":218,"threads_started":1}
I20260619 06:43:05.724556 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:05.723766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:43581)
0619 06:43:05.724261 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:05.724360 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":254,"threads_started":1}
I20260619 06:43:05.863903 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:05.863217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:41425)
0619 06:43:05.863655 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:05.863739 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":234,"threads_started":1}
I20260619 06:43:06.293677 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:06.292910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:52105)
0619 06:43:06.293404 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:06.293494 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":262,"threads_started":1}
I20260619 06:43:06.726449 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:06.725568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:42663)
0619 06:43:06.726107 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:06.726223 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":270,"threads_started":1}
I20260619 06:43:06.865803 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:06.864994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:36407)
0619 06:43:06.865505 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:06.865611 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":271,"threads_started":1}
I20260619 06:43:07.295326 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:07.294633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:37657)
0619 06:43:07.295073 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:07.295161 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":226,"threads_started":1}
I20260619 06:43:07.728295 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:07.727475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:57137)
0619 06:43:07.728011 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:07.728111 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":282,"threads_started":1}
I20260619 06:43:07.867628 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:07.866844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:45619)
0619 06:43:07.867331 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:07.867446 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":245,"threads_started":1}
I20260619 06:43:08.004916 23164 raft_consensus.cc:482] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:43:08.005139 23164 raft_consensus.cc:504] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:08.005985 23164 leader_election.cc:288] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377), 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867)
W20260619 06:43:08.006944 22765 leader_election.cc:341] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377): Not found: Tablet not found: 9f189f5a495447578703602e0f1ac347
W20260619 06:43:08.007109 22763 leader_election.cc:341] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867): Not found: Tablet not found: 9f189f5a495447578703602e0f1ac347
I20260619 06:43:08.007236 22763 leader_election.cc:302] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c00f0819a994c7fa1d68795889d3d7e; no voters: 2ff3edebbbb145d89b15575d6c8ee0dd, 3c98a2a5f0f4482da42b0cd37347fe1b
I20260619 06:43:08.007519 23164 raft_consensus.cc:2726] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:43:08.297206 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:08.296317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:57057)
0619 06:43:08.296887 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:08.296999 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":319,"threads_started":1}
I20260619 06:43:08.730136 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:08.729336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:49947)
0619 06:43:08.729861 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:08.729954 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":294,"threads_started":1}
I20260619 06:43:08.869334 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:08.868537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:57531)
0619 06:43:08.869040 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:08.869140 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":245,"threads_started":1}
I20260619 06:43:09.299166 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:09.298319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:51715)
0619 06:43:09.298868 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:09.298965 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":259,"threads_started":1}
I20260619 06:43:09.731945 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:09.731107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:58547)
0619 06:43:09.731672 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:09.731760 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":222,"threads_started":1}
I20260619 06:43:09.870996 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:09.870225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:55199)
0619 06:43:09.870733 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:09.870828 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":244,"threads_started":1}
I20260619 06:43:10.053153 23171 raft_consensus.cc:482] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:43:10.053368 23171 raft_consensus.cc:504] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:10.054023 23171 leader_election.cc:288] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377), 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867)
W20260619 06:43:10.054996 22763 leader_election.cc:341] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867): Not found: Tablet not found: 9f189f5a495447578703602e0f1ac347
W20260619 06:43:10.055176 22765 leader_election.cc:341] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377): Not found: Tablet not found: 9f189f5a495447578703602e0f1ac347
I20260619 06:43:10.055274 22765 leader_election.cc:302] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c00f0819a994c7fa1d68795889d3d7e; no voters: 2ff3edebbbb145d89b15575d6c8ee0dd, 3c98a2a5f0f4482da42b0cd37347fe1b
I20260619 06:43:10.055488 23171 raft_consensus.cc:2726] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:43:10.301038 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:10.300263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:37257)
0619 06:43:10.300765 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:10.300858 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":227,"threads_started":1}
I20260619 06:43:10.733567 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:10.732861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:51953)
0619 06:43:10.733285 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:10.733372 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":205,"threads_started":1}
I20260619 06:43:10.872862 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:10.871998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:52477)
0619 06:43:10.872489 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:10.872640 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":248,"threads_started":1}
I20260619 06:43:11.302804 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:11.302012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:40953)
0619 06:43:11.302535 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:11.302635 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":222,"threads_started":1}
I20260619 06:43:11.735280 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:11.734477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:39173)
0619 06:43:11.735018 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:11.735120 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":255,"threads_started":1}
I20260619 06:43:11.874534 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:11.873866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:40503)
0619 06:43:11.874281 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:11.874366 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260619 06:43:12.304657 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:12.303829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:36257)
0619 06:43:12.304315 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:12.304406 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":280,"threads_started":1}
I20260619 06:43:12.736995 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:12.736218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:58417)
0619 06:43:12.736721 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:12.736812 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":258,"threads_started":1}
I20260619 06:43:12.876170 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:12.875398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:37399)
0619 06:43:12.875852 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:12.875948 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":245,"threads_started":1}
I20260619 06:43:13.265846 23181 raft_consensus.cc:482] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:43:13.266036 23181 raft_consensus.cc:504] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff3edebbbb145d89b15575d6c8ee0dd" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 42377 } } peers { permanent_uuid: "2c00f0819a994c7fa1d68795889d3d7e" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44551 } } peers { permanent_uuid: "3c98a2a5f0f4482da42b0cd37347fe1b" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 40867 } }
I20260619 06:43:13.266788 23181 leader_election.cc:288] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377), 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867)
W20260619 06:43:13.267725 22765 leader_election.cc:341] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ff3edebbbb145d89b15575d6c8ee0dd (127.7.146.195:42377): Not found: Tablet not found: 9f189f5a495447578703602e0f1ac347
W20260619 06:43:13.267903 22763 leader_election.cc:341] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c98a2a5f0f4482da42b0cd37347fe1b (127.7.146.194:40867): Not found: Tablet not found: 9f189f5a495447578703602e0f1ac347
I20260619 06:43:13.268039 22763 leader_election.cc:302] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c00f0819a994c7fa1d68795889d3d7e; no voters: 2ff3edebbbb145d89b15575d6c8ee0dd, 3c98a2a5f0f4482da42b0cd37347fe1b
I20260619 06:43:13.268281 23181 raft_consensus.cc:2726] T 9f189f5a495447578703602e0f1ac347 P 2c00f0819a994c7fa1d68795889d3d7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:43:13.306360 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:13.305606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:55017)
0619 06:43:13.306109 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:13.306194 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":253,"threads_started":1}
I20260619 06:43:13.738543 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:13.737843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:40531)
0619 06:43:13.738301 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:13.738385 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":214,"threads_started":1}
I20260619 06:43:13.877915 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:13.877137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:46443)
0619 06:43:13.877616 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:13.877719 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":250,"threads_started":1}
I20260619 06:43:14.307945 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:14.307251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:56781)
0619 06:43:14.307665 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:14.307747 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":217,"threads_started":1}
I20260619 06:43:14.740252 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:14.739440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:45789)
0619 06:43:14.739927 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:14.740014 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":282,"threads_started":1}
I20260619 06:43:14.879580 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:14.878861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:39491)
0619 06:43:14.879321 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:14.879407 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":200,"threads_started":1}
I20260619 06:43:15.309697 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:15.308876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:47759)
0619 06:43:15.309421 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:15.309528 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":287,"threads_started":1}
I20260619 06:43:15.741910 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:15.741201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:40303)
0619 06:43:15.741637 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:15.741741 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":208,"threads_started":1}
I20260619 06:43:15.881198 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:15.880416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:50843)
0619 06:43:15.880915 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:15.881009 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":271,"threads_started":1}
I20260619 06:43:16.311353 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:16.310587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:37129)
0619 06:43:16.311057 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:16.311166 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":232,"threads_started":1}
I20260619 06:43:16.743500 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:16.742725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:45625)
0619 06:43:16.743215 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:16.743303 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":233,"threads_started":1}
I20260619 06:43:16.883239 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:16.882460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.193:52609)
0619 06:43:16.882939 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:16.883059 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":247,"threads_started":1}
I20260619 06:43:17.313337 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:17.312528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.194:37077)
0619 06:43:17.313058 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:17.313176 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":239,"threads_started":1}
I20260619 06:43:17.723204 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 22712
I20260619 06:43:17.742483 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:17.724734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.7.146.193:44551)
0619 06:43:17.725223 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:17.725233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:17.725266 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:43:17.725339 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:17.725348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:17.725677 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:17.725853 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:17.728077 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:17.729141 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:17.730359 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:17.730899 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:17.731000 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:17.736931 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:17.736945 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:17.736962 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:17.736999 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:17.739706 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:17.740415 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:17.740422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:17.740424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:17.740554 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:17.740946 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:17.740952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:17.740955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:17.741322 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:17.741620 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:17.741926 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 06:43:17.742195 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":1019,"threads_started":1}
I20260619 06:43:17.743680 22871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:17.745515 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:17.744655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40389 (local address 127.7.146.195:46221)
0619 06:43:17.745188 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:17.745334 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":293,"threads_started":1}
I20260619 06:43:17.834801 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 22712
I20260619 06:43:17.849013 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 22879
I20260619 06:43:17.862398 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:17.849764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.7.146.194:40867)
0619 06:43:17.850283 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:17.850287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:17.850302 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:43:17.850355 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:17.850358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:17.850422 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:17.850515 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:17.851866 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:17.852710 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:17.853934 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:17.854356 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:17.854462 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:17.857140 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:17.857145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:17.857148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:17.857181 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:17.859783 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:17.860469 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:17.860473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:17.860475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:17.860624 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:17.860964 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:17.860980 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:17.860984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:17.861254 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:17.861498 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:17.861800 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 06:43:17.862014 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":261,"threads_started":1}
I20260619 06:43:17.863070 23003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:17.946283 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 22879
I20260619 06:43:17.960727 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 23011
I20260619 06:43:17.973603 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:17.961562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.7.146.195:42377)
0619 06:43:17.962132 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:17.962138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:17.962155 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:43:17.962211 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:17.962215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:17.962284 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:17.962372 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:17.963719 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:17.964487 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:17.965815 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:17.966233 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:17.966324 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:17.968779 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:17.968783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:17.968787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:17.968816 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:17.971338 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:17.971999 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:17.972003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:17.972006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:17.972142 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:17.972420 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:17.972423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:17.972424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:17.972737 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:17.972941 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:17.973123 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 06:43:17.973288 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":292,"threads_started":1}
I20260619 06:43:17.974314 23134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:18.058416 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 23011
2026-06-19T06:43:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24009 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:43:18 dist-test-slave-qfk4 krb5kdc[23204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:43:18 dist-test-slave-qfk4 krb5kdc[23204](info): set up 2 sockets
Jun 19 06:43:18 dist-test-slave-qfk4 krb5kdc[23204](info): commencing operation
krb5kdc: starting...
W20260619 06:43:19.077006 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:43:19 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:43:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:43:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:43:19.314047 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34769
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:34769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:43:19.636083 23217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:19.636394 23217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:19.636525 23217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:19.643839 23217 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:43:19.643940 23217 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:43:19.643980 23217 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:43:19.644049 23217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:43:19.644115 23217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:19.644191 23217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:19.644245 23217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:43:19.644315 23217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:43:19.654438 23217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:34769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:19.655736 23217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:19.657056 23217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:19.668807 23223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:19.668859 23222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:19.668817 23225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:19.669494 23217 server_base.cc:1034] running on GCE node
I20260619 06:43:19.670327 23217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:19.671730 23217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:19.672923 23217 hybrid_clock.cc:648] HybridClock initialized: now 1781851399672850 us; error 51 us; skew 500 ppm
I20260619 06:43:19.673240 23217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:19 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:19.676292 23217 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:43:19.677774 23217 webserver.cc:460] Webserver started at http://127.7.146.254:43739/ using document root <none> and password file <none>
I20260619 06:43:19.678439 23217 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:19.678521 23217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:19.678819 23217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:43:19.681826 23217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "980d5b05b14a47cfb3b410f27a239569"
format_stamp: "Formatted at 2026-06-19 06:43:19 on dist-test-slave-qfk4"
server_key: "573cf799e85c9cbc16015e0c08e57737"
server_key_iv: "bd0307d8b7c690e9897e8ac3e317c6df"
server_key_version: "encryptionkey@0"
I20260619 06:43:19.682569 23217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "980d5b05b14a47cfb3b410f27a239569"
format_stamp: "Formatted at 2026-06-19 06:43:19 on dist-test-slave-qfk4"
server_key: "573cf799e85c9cbc16015e0c08e57737"
server_key_iv: "bd0307d8b7c690e9897e8ac3e317c6df"
server_key_version: "encryptionkey@0"
I20260619 06:43:19.687716 23217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:43:19.691391 23232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:19.692906 23217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:43:19.693056 23217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "980d5b05b14a47cfb3b410f27a239569"
format_stamp: "Formatted at 2026-06-19 06:43:19 on dist-test-slave-qfk4"
server_key: "573cf799e85c9cbc16015e0c08e57737"
server_key_iv: "bd0307d8b7c690e9897e8ac3e317c6df"
server_key_version: "encryptionkey@0"
I20260619 06:43:19.693251 23217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:19.712289 23217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:19.716454 23217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:19.716725 23217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:19.734017 23217 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:34769
I20260619 06:43:19.734055 23283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:34769 every 8 connection(s)
I20260619 06:43:19.735311 23217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:43:19.739060 23284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:43:19.744768 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 23217
I20260619 06:43:19.744956 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:43:19.745607 7755 external_mini_cluster.cc:1428] Setting key 7d16ddb3c276b6963c2b742622cf5d1d
I20260619 06:43:19.749807 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Bootstrap starting.
I20260619 06:43:19.753437 23284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Neither blocks nor log segments found. Creating new log.
I20260619 06:43:19.754596 23284 log.cc:824] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:19.757675 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: No bootstrap required, opened a new log
Jun 19 06:43:19 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:43:19.765944 23284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:43:19.766326 23284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:43:19.766423 23284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980d5b05b14a47cfb3b410f27a239569, State: Initialized, Role: FOLLOWER
I20260619 06:43:19.767163 23284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:43:19.767397 23284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:43:19.767527 23284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:43:19.767700 23284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:43:19.768266 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:19.747585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.7.146.254:34769)
0619 06:43:19.748343 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:19.748355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:19.748397 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:43:19.750116 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:19.750138 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:19.750532 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:19.750861 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:19.752697 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:19.754055 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:19.755701 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:19.756379 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:19.756513 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:19.761120 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:19.761129 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:19.761145 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:19.761185 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:19.764120 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:19.764835 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:19.764842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:19.764849 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:19.765026 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:19.765412 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:19.765416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:19.765418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:19.765975 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:19.766253 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:19.766714 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 06:43:19.767121 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":343,"threads_started":1}
I20260619 06:43:19.770149 23284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:43:19.770659 23284 leader_election.cc:302] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 980d5b05b14a47cfb3b410f27a239569; no voters:
I20260619 06:43:19.771284 23284 leader_election.cc:288] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:43:19.771354 23289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:43:19.771977 23289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 1 LEADER]: Becoming Leader. State: Replica: 980d5b05b14a47cfb3b410f27a239569, State: Running, Role: LEADER
I20260619 06:43:19.772542 23289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:43:19.773475 23284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:43:19.775651 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 980d5b05b14a47cfb3b410f27a239569. Latest consensus state: current_term: 1 leader_uuid: "980d5b05b14a47cfb3b410f27a239569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } } }
I20260619 06:43:19.775704 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "980d5b05b14a47cfb3b410f27a239569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } } }
I20260619 06:43:19.776021 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: This master's current role is: LEADER
I20260619 06:43:19.776042 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: This master's current role is: LEADER
I20260619 06:43:19.779795 23296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:43:19.784991 23296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:43:19.785686 23306 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:43:19.794564 23296 catalog_manager.cc:1347] Generated new cluster ID: d2a32ea2731441d6b3367409b3f048fd
I20260619 06:43:19.794695 23296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:43:19.815436 23296 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:43:19.817008 23296 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:43:19.828723 23296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Generated new TSK 0
I20260619 06:43:19.829799 23296 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:43:19.887658 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34769
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:43:20.214013 23310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:20.214357 23310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:20.214459 23310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:20.221930 23310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:20.222048 23310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:20.222168 23310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:43:20.232071 23310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:20.233371 23310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:20.234683 23310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:20.247490 23315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:20.247568 23318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:20.247547 23316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:20.248095 23310 server_base.cc:1034] running on GCE node
I20260619 06:43:20.248752 23310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:20.249660 23310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:20.250869 23310 hybrid_clock.cc:648] HybridClock initialized: now 1781851400250826 us; error 34 us; skew 500 ppm
I20260619 06:43:20.251189 23310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:20 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851400, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:20.254251 23310 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:43:20.255681 23310 webserver.cc:460] Webserver started at http://127.7.146.193:42887/ using document root <none> and password file <none>
I20260619 06:43:20.256317 23310 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:20.256433 23310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:20.256778 23310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:43:20.259629 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "08342fc6c0eb48cb98716f9ed8eaf287"
format_stamp: "Formatted at 2026-06-19 06:43:20 on dist-test-slave-qfk4"
server_key: "3e134de4600882f47b1164dbbdfb7d5e"
server_key_iv: "9e9178ac6531e79b692caf5e48c15973"
server_key_version: "encryptionkey@0"
I20260619 06:43:20.260396 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "08342fc6c0eb48cb98716f9ed8eaf287"
format_stamp: "Formatted at 2026-06-19 06:43:20 on dist-test-slave-qfk4"
server_key: "3e134de4600882f47b1164dbbdfb7d5e"
server_key_iv: "9e9178ac6531e79b692caf5e48c15973"
server_key_version: "encryptionkey@0"
I20260619 06:43:20.265580 23310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:43:20.268977 23325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:20.270323 23310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:43:20.270483 23310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "08342fc6c0eb48cb98716f9ed8eaf287"
format_stamp: "Formatted at 2026-06-19 06:43:20 on dist-test-slave-qfk4"
server_key: "3e134de4600882f47b1164dbbdfb7d5e"
server_key_iv: "9e9178ac6531e79b692caf5e48c15973"
server_key_version: "encryptionkey@0"
I20260619 06:43:20.270680 23310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:20.291803 23310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:20.296077 23310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:20.296340 23310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:20.297410 23310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:43:20.298763 23310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:43:20.298846 23310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:20.298964 23310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:43:20.299015 23310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:20.329838 23310 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:35943
I20260619 06:43:20.329907 23437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:35943 every 8 connection(s)
I20260619 06:43:20.331091 23310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:43:20.339130 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 23310
I20260619 06:43:20.339325 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:43:20.339972 7755 external_mini_cluster.cc:1428] Setting key 143967ce4a22a8de513b4ef197d15774
Jun 19 06:43:20 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851400, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:43:20.351045 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:20.333526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:60921 (local address 127.7.146.254:34769)
0619 06:43:20.333951 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:20.333956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:20.334853 (+ 897us) server_negotiation.cc:407] Connection header received
0619 06:43:20.336711 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:20.336718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:20.336798 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:20.336893 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:20.338385 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:20.339111 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:20.340810 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:20.341306 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:20.341421 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:20.345110 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:20.345116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:20.345121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:20.345161 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:20.348081 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:20.348824 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:20.348830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:20.348833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:20.348969 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:20.349336 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:20.349341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:20.349344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:20.349662 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:20.349794 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:20.350628 (+ 834us) server_negotiation.cc:299] Negotiation successful
0619 06:43:20.350798 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":245,"threads_started":1}
I20260619 06:43:20.352376 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:20.334029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:60921)
0619 06:43:20.334700 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:20.334741 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:20.336434 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:20.337142 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:20.337151 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:20.337657 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:43:20.338214 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:20.338232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:20.339266 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:20.339272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:20.340641 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:20.340656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:20.341526 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:20.341533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:20.341828 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:20.342567 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:43:20.342590 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:43:20.344880 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:43:20.348226 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:20.348233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:20.348247 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:20.348711 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:20.349061 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:20.349064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:20.349066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:20.349241 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:20.349773 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:43:20.349780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:43:20.350212 (+ 432us) client_negotiation.cc:769] Sending connection context
0619 06:43:20.350504 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 06:43:20.351054 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":249,"threads_started":1}
I20260619 06:43:20.353888 23438 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34769
I20260619 06:43:20.354187 23438 heartbeater.cc:464] Registering TS with master...
I20260619 06:43:20.354846 23438 heartbeater.cc:511] Master 127.7.146.254:34769 requested a full tablet report, sending...
I20260619 06:43:20.356897 23249 ts_manager.cc:194] Registered new tserver with Master: 08342fc6c0eb48cb98716f9ed8eaf287 (127.7.146.193:35943)
I20260619 06:43:20.358372 23249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:60921
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:43:20.389899 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34769
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:43:20.705199 23444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:20.705492 23444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:20.705629 23444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:20.712875 23444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:20.712993 23444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:20.713141 23444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:43:20.722976 23444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:20.724228 23444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:20.725492 23444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:20.739142 23449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:20.739169 23450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:20.740114 23452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:20.740500 23444 server_base.cc:1034] running on GCE node
I20260619 06:43:20.741149 23444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:20.742108 23444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:20.743335 23444 hybrid_clock.cc:648] HybridClock initialized: now 1781851400743274 us; error 53 us; skew 500 ppm
I20260619 06:43:20.743692 23444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:20 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851400, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:20.746920 23444 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:43:20.748471 23444 webserver.cc:460] Webserver started at http://127.7.146.194:40263/ using document root <none> and password file <none>
I20260619 06:43:20.749157 23444 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:20.749274 23444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:20.749553 23444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:43:20.752404 23444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "cf10a2ea76e34c4d9599fe61f7e7777e"
format_stamp: "Formatted at 2026-06-19 06:43:20 on dist-test-slave-qfk4"
server_key: "41320e298804df891289125d42219d2a"
server_key_iv: "96ade8f2197917ac8b1c389cdb754ec2"
server_key_version: "encryptionkey@0"
I20260619 06:43:20.753221 23444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "cf10a2ea76e34c4d9599fe61f7e7777e"
format_stamp: "Formatted at 2026-06-19 06:43:20 on dist-test-slave-qfk4"
server_key: "41320e298804df891289125d42219d2a"
server_key_iv: "96ade8f2197917ac8b1c389cdb754ec2"
server_key_version: "encryptionkey@0"
I20260619 06:43:20.758399 23444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:43:20.761968 23459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:20.763464 23444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:43:20.763630 23444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "cf10a2ea76e34c4d9599fe61f7e7777e"
format_stamp: "Formatted at 2026-06-19 06:43:20 on dist-test-slave-qfk4"
server_key: "41320e298804df891289125d42219d2a"
server_key_iv: "96ade8f2197917ac8b1c389cdb754ec2"
server_key_version: "encryptionkey@0"
I20260619 06:43:20.763826 23444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:20.803160 23444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:20.807350 23444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:20.807616 23444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:20.808641 23444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:43:20.809973 23444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:43:20.810065 23444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:20.810181 23444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:43:20.810232 23444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:20.839656 23444 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:39989
I20260619 06:43:20.839704 23571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:39989 every 8 connection(s)
I20260619 06:43:20.840917 23444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:43:20.842118 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 23444
I20260619 06:43:20.842298 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:43:20.842875 7755 external_mini_cluster.cc:1428] Setting key 6b182403a22ef5a338a33877680bb700
Jun 19 06:43:20 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851400, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:43:20.861922 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:20.843706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:38153 (local address 127.7.146.254:34769)
0619 06:43:20.843923 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:20.843929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:20.845241 (+ 1312us) server_negotiation.cc:407] Connection header received
0619 06:43:20.847073 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:20.847078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:20.847159 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:20.847252 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:20.848821 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:20.849781 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:20.851323 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:20.851767 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:20.851864 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:20.855866 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:20.855872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:20.855876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:20.855913 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:20.858472 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:20.859455 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:20.859460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:20.859462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:20.859596 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:20.860072 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:20.860076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:20.860077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:20.860365 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:20.860499 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:20.861296 (+ 797us) server_negotiation.cc:299] Negotiation successful
0619 06:43:20.861567 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:43:20.862994 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:20.844287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:38153)
0619 06:43:20.845074 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:20.845120 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:20.846811 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:20.847461 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:20.847470 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:20.848028 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:43:20.848660 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:20.848683 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:20.849924 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:20.849927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:20.851183 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:20.851195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:20.851986 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:20.851992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:20.852309 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:20.853024 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:43:20.853048 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:43:20.855575 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:43:20.858633 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:20.858643 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:20.858660 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:20.859307 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:20.859727 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:20.859732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:20.859736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:20.859977 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:20.860502 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:43:20.860510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:43:20.860914 (+ 404us) client_negotiation.cc:769] Sending connection context
0619 06:43:20.861198 (+ 284us) client_negotiation.cc:240] Negotiation successful
0619 06:43:20.861800 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":344,"threads_started":1}
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
I20260619 06:43:20.864441 23572 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34769
I20260619 06:43:20.864822 23572 heartbeater.cc:464] Registering TS with master...
I20260619 06:43:20.865592 23572 heartbeater.cc:511] Master 127.7.146.254:34769 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
I20260619 06:43:20.867030 23249 ts_manager.cc:194] Registered new tserver with Master: cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989)
I20260619 06:43:20.867969 23249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:38153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:43:20.892071 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34769
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:43:21.209028 23577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:21.209303 23577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:21.209398 23577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:21.216507 23577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:21.216655 23577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:21.216784 23577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:43:21.226768 23577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:21.228003 23577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:21.229290 23577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:21.242066 23582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:21.242193 23583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:21.242139 23585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:21.243070 23577 server_base.cc:1034] running on GCE node
I20260619 06:43:21.243677 23577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:21.244675 23577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:21.245863 23577 hybrid_clock.cc:648] HybridClock initialized: now 1781851401245819 us; error 40 us; skew 500 ppm
I20260619 06:43:21.246173 23577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:21 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851401, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:21.249203 23577 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:43:21.250676 23577 webserver.cc:460] Webserver started at http://127.7.146.195:37443/ using document root <none> and password file <none>
I20260619 06:43:21.251335 23577 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:21.251448 23577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:21.251746 23577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:43:21.254733 23577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "d541cd83981940c5b7df8fe745e949d0"
format_stamp: "Formatted at 2026-06-19 06:43:21 on dist-test-slave-qfk4"
server_key: "5b00766cb0d2fa76e7b9692777694854"
server_key_iv: "4826f1a13b7cb4c02285cab39bda7b14"
server_key_version: "encryptionkey@0"
I20260619 06:43:21.255465 23577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "d541cd83981940c5b7df8fe745e949d0"
format_stamp: "Formatted at 2026-06-19 06:43:21 on dist-test-slave-qfk4"
server_key: "5b00766cb0d2fa76e7b9692777694854"
server_key_iv: "4826f1a13b7cb4c02285cab39bda7b14"
server_key_version: "encryptionkey@0"
I20260619 06:43:21.260703 23577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:43:21.264398 23592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:21.265982 23577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:43:21.266167 23577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "d541cd83981940c5b7df8fe745e949d0"
format_stamp: "Formatted at 2026-06-19 06:43:21 on dist-test-slave-qfk4"
server_key: "5b00766cb0d2fa76e7b9692777694854"
server_key_iv: "4826f1a13b7cb4c02285cab39bda7b14"
server_key_version: "encryptionkey@0"
I20260619 06:43:21.266381 23577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:21.288923 23577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:21.293361 23577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:21.293640 23577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:21.294656 23577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:43:21.295997 23577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:43:21.296080 23577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:21.296216 23577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:43:21.296265 23577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:21.326939 23577 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:43567
I20260619 06:43:21.327013 23704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:43567 every 8 connection(s)
I20260619 06:43:21.328428 23577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:43:21.335583 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 23577
I20260619 06:43:21.335738 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:43:21.336145 7755 external_mini_cluster.cc:1428] Setting key 712a5c469af8d05ccd93430d5d43627e
Jun 19 06:43:21 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851401, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:43:21.348804 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.331189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:50671 (local address 127.7.146.254:34769)
0619 06:43:21.331406 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:21.331413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:21.332685 (+ 1272us) server_negotiation.cc:407] Connection header received
0619 06:43:21.334567 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:21.334574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:21.334672 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:21.334768 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:21.336314 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.337244 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.339328 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.339842 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.339937 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.343239 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:21.343251 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:21.343257 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:21.343302 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:21.345880 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:21.346673 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:21.346678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:21.346680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:21.346810 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:21.347216 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:21.347220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:21.347222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:21.347506 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:21.347625 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:21.348300 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 06:43:21.348466 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:43:21.349853 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.331699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:50671)
0619 06:43:21.332484 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:21.332545 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:21.334302 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:21.334964 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:21.334973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:21.335520 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:43:21.336133 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.336151 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.337405 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.337411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.339172 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.339187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.340034 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.340039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.340248 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.340900 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:43:21.340923 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:43:21.343014 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:43:21.346029 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:21.346037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:21.346051 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:21.346552 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:21.346906 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:21.346909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:21.346912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:21.347111 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:21.347628 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:43:21.347634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:43:21.347961 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 06:43:21.348201 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 06:43:21.348680 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":314,"threads_started":1}
I20260619 06:43:21.351202 23705 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34769
I20260619 06:43:21.351514 23705 heartbeater.cc:464] Registering TS with master...
I20260619 06:43:21.352219 23705 heartbeater.cc:511] Master 127.7.146.254:34769 requested a full tablet report, sending...
I20260619 06:43:21.353611 23249 ts_manager.cc:194] Registered new tserver with Master: d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
I20260619 06:43:21.354430 23249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:50671
I20260619 06:43:21.361231 23438 heartbeater.cc:503] Master 127.7.146.254:34769 was elected leader, sending a full tablet report...
I20260619 06:43:21.363945 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:43:21.380900 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.368361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.7.146.254:34769)
0619 06:43:21.368534 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:21.368539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:21.368919 (+ 380us) server_negotiation.cc:407] Connection header received
0619 06:43:21.369057 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:21.369060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:21.369122 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:21.369199 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:21.370733 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.371510 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.373073 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.373513 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.373650 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.376070 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:21.376075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:21.376078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:21.376109 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:21.378563 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:21.379172 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:21.379177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:21.379181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:21.379321 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:21.379670 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:21.379674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:21.379676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:21.379993 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:21.380179 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:21.380478 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 06:43:21.380679 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:43:21.384994 23248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:43:21.391701 23248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:43:21.415802 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.403945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.7.146.194:39989)
0619 06:43:21.406364 (+ 2419us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:21.406374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:21.406403 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:43:21.406501 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:21.406509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:21.406781 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:21.408270 (+ 1489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:21.408495 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.410094 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.412603 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.414052 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.414199 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.414406 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:21.415272 (+ 866us) server_negotiation.cc:299] Negotiation successful
0619 06:43:21.415463 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2244,"thread_start_us":2107,"threads_started":1}
I20260619 06:43:21.415850 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.403758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:39989 (local address 127.0.0.1:52100)
0619 06:43:21.405411 (+ 1653us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:21.405456 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:21.405679 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:21.407222 (+ 1543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:21.407230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:21.407263 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:21.408068 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.408083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.410234 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.410240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.412419 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.412432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.414392 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.414402 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.414640 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.414698 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 06:43:21.414830 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 06:43:21.415036 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1377,"thread_start_us":2698,"threads_started":1}
I20260619 06:43:21.419715 23507 tablet_service.cc:1467] Processing CreateTablet for tablet 70cee97e14d442fbbfd025ea881abaf6 (DEFAULT_TABLE table=test-table [id=122b12c36485406686be03512081bd0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:43:21.420816 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.407541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:43567 (local address 127.0.0.1:59232)
0619 06:43:21.412791 (+ 5250us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:21.412809 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:21.412909 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:21.414027 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:21.414032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:21.414058 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:21.414420 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.414433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.416315 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.416320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.417998 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.418011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.419855 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.419862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.420037 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.420100 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 06:43:21.420223 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 06:43:21.420362 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5117,"mutex_wait_us":25,"thread_start_us":2860,"threads_started":1}
I20260619 06:43:21.420953 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.407133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.7.146.195:43567)
0619 06:43:21.413434 (+ 6301us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:21.413443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:21.413473 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:43:21.413562 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:21.413569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:21.413821 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:21.414018 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:21.414616 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.416152 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.418164 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.419626 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.419771 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.420016 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:21.420403 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 06:43:21.420604 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6124,"thread_start_us":5968,"threads_started":1}
I20260619 06:43:21.421402 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.404820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.7.146.193:35943)
0619 06:43:21.412115 (+ 7295us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:21.412125 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:21.412159 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:43:21.412273 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:21.412283 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:21.412557 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:21.412810 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:21.413638 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.414984 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.418249 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.419703 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.419827 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.420025 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:21.420930 (+ 905us) server_negotiation.cc:299] Negotiation successful
0619 06:43:21.421098 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7083,"thread_start_us":6914,"threads_started":1}
I20260619 06:43:21.421931 23717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.404167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:46858)
0619 06:43:21.406748 (+ 2581us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:21.406769 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:21.406889 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:21.413085 (+ 6196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:21.413091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:21.413120 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:21.413459 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.413469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.415145 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.415150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.418090 (+ 2940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.418106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.420177 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.420184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.420351 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.420383 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:43:21.420464 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 06:43:21.420624 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2439,"mutex_wait_us":22,"thread_start_us":831,"threads_started":1}
I20260619 06:43:21.422289 23507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70cee97e14d442fbbfd025ea881abaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:43:21.424937 23373 tablet_service.cc:1467] Processing CreateTablet for tablet 70cee97e14d442fbbfd025ea881abaf6 (DEFAULT_TABLE table=test-table [id=122b12c36485406686be03512081bd0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:43:21.425122 23640 tablet_service.cc:1467] Processing CreateTablet for tablet 70cee97e14d442fbbfd025ea881abaf6 (DEFAULT_TABLE table=test-table [id=122b12c36485406686be03512081bd0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:43:21.426173 23373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70cee97e14d442fbbfd025ea881abaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:43:21.426554 23640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70cee97e14d442fbbfd025ea881abaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:43:21.437412 23723 tablet_bootstrap.cc:492] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Bootstrap starting.
I20260619 06:43:21.440551 23723 tablet_bootstrap.cc:654] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Neither blocks nor log segments found. Creating new log.
I20260619 06:43:21.441323 23724 tablet_bootstrap.cc:492] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: Bootstrap starting.
I20260619 06:43:21.442013 23723 log.cc:824] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:21.443040 23725 tablet_bootstrap.cc:492] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: Bootstrap starting.
I20260619 06:43:21.445016 23724 tablet_bootstrap.cc:654] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: Neither blocks nor log segments found. Creating new log.
I20260619 06:43:21.446192 23724 log.cc:824] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:21.446530 23725 tablet_bootstrap.cc:654] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: Neither blocks nor log segments found. Creating new log.
I20260619 06:43:21.447363 23723 tablet_bootstrap.cc:492] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: No bootstrap required, opened a new log
I20260619 06:43:21.447641 23723 ts_tablet_manager.cc:1397] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 06:43:21.447685 23725 log.cc:824] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:21.449359 23724 tablet_bootstrap.cc:492] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: No bootstrap required, opened a new log
I20260619 06:43:21.449633 23724 ts_tablet_manager.cc:1397] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:43:21.450398 23725 tablet_bootstrap.cc:492] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: No bootstrap required, opened a new log
I20260619 06:43:21.450668 23725 ts_tablet_manager.cc:1397] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 06:43:21.458312 23725 raft_consensus.cc:348] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } }
I20260619 06:43:21.458336 23723 raft_consensus.cc:348] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } }
I20260619 06:43:21.458720 23725 raft_consensus.cc:374] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:43:21.458818 23723 raft_consensus.cc:374] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:43:21.458833 23725 raft_consensus.cc:729] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d541cd83981940c5b7df8fe745e949d0, State: Initialized, Role: FOLLOWER
I20260619 06:43:21.458918 23723 raft_consensus.cc:729] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08342fc6c0eb48cb98716f9ed8eaf287, State: Initialized, Role: FOLLOWER
I20260619 06:43:21.459439 23725 consensus_queue.cc:260] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } }
I20260619 06:43:21.459587 23723 consensus_queue.cc:260] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } }
I20260619 06:43:21.459980 23724 raft_consensus.cc:348] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } }
I20260619 06:43:21.460517 23724 raft_consensus.cc:374] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:43:21.460644 23705 heartbeater.cc:503] Master 127.7.146.254:34769 was elected leader, sending a full tablet report...
I20260619 06:43:21.460700 23724 raft_consensus.cc:729] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf10a2ea76e34c4d9599fe61f7e7777e, State: Initialized, Role: FOLLOWER
I20260619 06:43:21.461408 23724 consensus_queue.cc:260] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } }
I20260619 06:43:21.461625 23725 ts_tablet_manager.cc:1428] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: Time spent starting tablet: real 0.011s user 0.001s sys 0.011s
I20260619 06:43:21.466042 23572 heartbeater.cc:503] Master 127.7.146.254:34769 was elected leader, sending a full tablet report...
I20260619 06:43:21.467705 23723 ts_tablet_manager.cc:1428] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Time spent starting tablet: real 0.020s user 0.013s sys 0.008s
I20260619 06:43:21.467991 23724 ts_tablet_manager.cc:1428] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260619 06:43:21.580233 23706 tablet.cc:2369] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:43:21.583321 23439 tablet.cc:2369] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:43:21.592892 23573 tablet.cc:2369] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:43:21.732208 23730 raft_consensus.cc:482] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:43:21.732586 23730 raft_consensus.cc:504] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } }
I20260619 06:43:21.734639 23730 leader_election.cc:288] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989), d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
I20260619 06:43:21.741436 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.734992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:43545 (local address 127.7.146.194:39989)
0619 06:43:21.735213 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:21.735221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:21.735789 (+ 568us) server_negotiation.cc:407] Connection header received
0619 06:43:21.736045 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:21.736053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:21.736146 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:21.736210 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:21.736924 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.737977 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.739780 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.740589 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.740737 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.740813 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:21.741069 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 06:43:21.741194 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:43:21.741479 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.735015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:39989 (local address 127.7.146.193:43545)
0619 06:43:21.735696 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:21.735716 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:21.735856 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:21.736412 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:21.736416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:21.736438 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:21.736783 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.736796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.738111 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.738124 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.739638 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.739649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.740790 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.740796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.740931 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.740955 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:43:21.741034 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:43:21.741167 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":353,"threads_started":1}
I20260619 06:43:21.742832 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.736033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:37515 (local address 127.7.146.195:43567)
0619 06:43:21.736212 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:21.736217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:21.736651 (+ 434us) server_negotiation.cc:407] Connection header received
0619 06:43:21.736857 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:21.736862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:21.736937 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:21.737010 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:21.737504 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.738279 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.740778 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.741984 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.742132 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.742201 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:21.742474 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 06:43:21.742600 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:43:21.742864 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.735879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:43567 (local address 127.7.146.193:37515)
0619 06:43:21.736522 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:21.736543 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:21.736719 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:21.737040 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:21.737044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:21.737066 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:21.737363 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.737371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.738428 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.738433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.740602 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:21.740617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.742172 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.742178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:21.742331 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.742355 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:43:21.742432 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:43:21.742546 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":339,"threads_started":1}
I20260619 06:43:21.742784 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cee97e14d442fbbfd025ea881abaf6" candidate_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" is_pre_election: true
I20260619 06:43:21.743328 23527 raft_consensus.cc:2454] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08342fc6c0eb48cb98716f9ed8eaf287 in term 0.
I20260619 06:43:21.743880 23660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cee97e14d442fbbfd025ea881abaf6" candidate_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d541cd83981940c5b7df8fe745e949d0" is_pre_election: true
I20260619 06:43:21.744029 23328 leader_election.cc:302] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08342fc6c0eb48cb98716f9ed8eaf287, cf10a2ea76e34c4d9599fe61f7e7777e; no voters:
I20260619 06:43:21.744293 23660 raft_consensus.cc:2454] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08342fc6c0eb48cb98716f9ed8eaf287 in term 0.
I20260619 06:43:21.744438 23730 raft_consensus.cc:2781] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:43:21.744603 23730 raft_consensus.cc:482] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:43:21.744714 23730 raft_consensus.cc:3037] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:43:21.746809 23730 raft_consensus.cc:504] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } }
I20260619 06:43:21.747386 23730 leader_election.cc:288] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 1 election: Requested vote from peers cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989), d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
I20260619 06:43:21.747892 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cee97e14d442fbbfd025ea881abaf6" candidate_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e"
I20260619 06:43:21.747943 23660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cee97e14d442fbbfd025ea881abaf6" candidate_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d541cd83981940c5b7df8fe745e949d0"
I20260619 06:43:21.748122 23527 raft_consensus.cc:3037] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:43:21.748122 23660 raft_consensus.cc:3037] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:43:21.751256 23660 raft_consensus.cc:2454] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08342fc6c0eb48cb98716f9ed8eaf287 in term 1.
I20260619 06:43:21.751253 23527 raft_consensus.cc:2454] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08342fc6c0eb48cb98716f9ed8eaf287 in term 1.
I20260619 06:43:21.751683 23328 leader_election.cc:302] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08342fc6c0eb48cb98716f9ed8eaf287, cf10a2ea76e34c4d9599fe61f7e7777e; no voters:
I20260619 06:43:21.751950 23730 raft_consensus.cc:2781] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:43:21.752406 23730 raft_consensus.cc:686] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 LEADER]: Becoming Leader. State: Replica: 08342fc6c0eb48cb98716f9ed8eaf287, State: Running, Role: LEADER
I20260619 06:43:21.753129 23730 consensus_queue.cc:237] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } }
I20260619 06:43:21.758023 23249 catalog_manager.cc:5495] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08342fc6c0eb48cb98716f9ed8eaf287 (127.7.146.193). New cstate: current_term: 1 leader_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } health_report { overall_health: HEALTHY } } }
I20260619 06:43:21.781937 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.776550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.7.146.193:35943)
0619 06:43:21.776847 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:21.776854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:21.776876 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:43:21.777080 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:21.777087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:21.777184 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:21.777299 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:43:21.777647 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.778348 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.779811 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.780307 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.780407 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.780949 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:43:21.781457 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:43:21.781621 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:21.781662 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 06:43:21.781730 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 06:43:21.789896 23660 raft_consensus.cc:1261] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0 [term 1 FOLLOWER]: Refusing update from remote peer 08342fc6c0eb48cb98716f9ed8eaf287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:43:21.789942 23527 raft_consensus.cc:1261] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e [term 1 FOLLOWER]: Refusing update from remote peer 08342fc6c0eb48cb98716f9ed8eaf287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:43:21.790792 23734 consensus_queue.cc:1035] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:43:21.791115 23730 consensus_queue.cc:1035] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:43:21.811259 23738 mvcc.cc:205] Tried to move back new op lower bound from 7298463341721837568 to 7298463341584560128. Current Snapshot: MvccSnapshot[applied={T|T < 7298463341721837568}]
I20260619 06:43:21.819166 23248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:43:21.819434 23248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:43:21.822865 23248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Sending DeleteTablet for 3 replicas of tablet 70cee97e14d442fbbfd025ea881abaf6
I20260619 06:43:21.824139 23507 tablet_service.cc:1514] Processing DeleteTablet for tablet 70cee97e14d442fbbfd025ea881abaf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:43:21 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:52100
I20260619 06:43:21.824412 23373 tablet_service.cc:1514] Processing DeleteTablet for tablet 70cee97e14d442fbbfd025ea881abaf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:43:21 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:46858
I20260619 06:43:21.824493 23640 tablet_service.cc:1514] Processing DeleteTablet for tablet 70cee97e14d442fbbfd025ea881abaf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:43:21 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:59232
I20260619 06:43:21.825404 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 23310
I20260619 06:43:21.852900 23747 tablet_replica.cc:330] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: stopping tablet replica
I20260619 06:43:21.853390 23747 raft_consensus.cc:2229] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:43:21.853893 23747 raft_consensus.cc:2258] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:43:21.860735 23747 ts_tablet_manager.cc:1905] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:43:21.865876 23748 tablet_replica.cc:330] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: stopping tablet replica
I20260619 06:43:21.866324 23746 tablet_replica.cc:330] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: stopping tablet replica
I20260619 06:43:21.866819 23746 raft_consensus.cc:2229] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:43:21.867355 23746 raft_consensus.cc:2258] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:43:21.868734 23747 ts_tablet_manager.cc:1918] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:43:21.868957 23747 log.cc:1196] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/70cee97e14d442fbbfd025ea881abaf6
I20260619 06:43:21.869025 23748 raft_consensus.cc:2229] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:43:21.869534 23747 ts_tablet_manager.cc:1939] T 70cee97e14d442fbbfd025ea881abaf6 P d541cd83981940c5b7df8fe745e949d0: Deleting consensus metadata
Jun 19 06:43:21 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:43:21.871134 23748 raft_consensus.cc:2258] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:43:21.871541 23235 catalog_manager.cc:4841] TS d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567): tablet 70cee97e14d442fbbfd025ea881abaf6 (table test-table [id=122b12c36485406686be03512081bd0a]) successfully deleted
I20260619 06:43:21.873899 23746 ts_tablet_manager.cc:1905] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:43:21.875785 23748 ts_tablet_manager.cc:1905] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:43:21.880977 23746 ts_tablet_manager.cc:1918] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:43:21.881176 23746 log.cc:1196] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/70cee97e14d442fbbfd025ea881abaf6
I20260619 06:43:21.881707 23746 ts_tablet_manager.cc:1939] T 70cee97e14d442fbbfd025ea881abaf6 P cf10a2ea76e34c4d9599fe61f7e7777e: Deleting consensus metadata
I20260619 06:43:21.882660 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.826569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.7.146.193:35943)
0619 06:43:21.826753 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:21.826760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:21.849775 (+ 23015us) server_negotiation.cc:407] Connection header received
0619 06:43:21.850260 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:21.850266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:21.850346 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:21.850411 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:21.854274 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.855477 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.858091 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:21.858830 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:21.858949 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:21.874238 (+ 15289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:21.874253 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:21.874269 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:21.874315 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:21.878453 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:21.879474 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:21.879486 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:21.879490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:21.879816 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:21.880328 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:21.880333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:21.880341 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:21.880886 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:21.881373 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:21.882084 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 06:43:21.882299 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:43:21.884099 23433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:21.885285 23235 catalog_manager.cc:4841] TS cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989): tablet 70cee97e14d442fbbfd025ea881abaf6 (table test-table [id=122b12c36485406686be03512081bd0a]) successfully deleted
I20260619 06:43:21.983043 23748 ts_tablet_manager.cc:1918] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:43:21.983341 23748 log.cc:1196] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/70cee97e14d442fbbfd025ea881abaf6
I20260619 06:43:21.983762 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 23310
I20260619 06:43:21.983924 23748 ts_tablet_manager.cc:1939] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Deleting consensus metadata
W20260619 06:43:21.998533 23234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.146.193:35943 (error 108)
I20260619 06:43:21.998708 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 23444
I20260619 06:43:22.000735 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.999872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:46886)
0619 06:43:22.000299 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:22.000424 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":209,"threads_started":1}
W20260619 06:43:22.001015 23234 catalog_manager.cc:4584] TS 08342fc6c0eb48cb98716f9ed8eaf287 (127.7.146.193:35943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70cee97e14d442fbbfd025ea881abaf6: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Jun 19 06:43:22 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:43:22.014915 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:21.999589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.7.146.194:39989)
0619 06:43:21.999788 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:21.999795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:21.999819 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:43:21.999991 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:21.999998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:22.000111 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:22.000245 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:22.001768 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:22.002478 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:22.003903 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:22.004308 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:22.004405 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:22.008674 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:22.008693 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:22.008723 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:22.008775 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:22.011806 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:22.012731 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:22.012740 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:22.012742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:22.012901 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:22.013346 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:22.013350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:22.013352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:22.013795 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:22.014065 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:22.014370 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 06:43:22.014616 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:43:22.015676 23567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:22.054728 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:22.054162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:46888)
0619 06:43:22.054408 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:22.054511 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260619 06:43:22.055850 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:22.055424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:46894)
0619 06:43:22.055624 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:22.055703 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 06:43:22.105365 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 23444
I20260619 06:43:22.116639 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:22.116048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:46896)
0619 06:43:22.116306 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:22.116430 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 06:43:22.117760 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:22.117337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:46906)
0619 06:43:22.117487 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:22.117587 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 06:43:22.118999 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 23577
Jun 19 06:43:22 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:43:22.134936 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:22.119743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.7.146.195:43567)
0619 06:43:22.119893 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:22.119898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:22.119945 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:43:22.120142 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:22.120147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:22.120222 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:22.120300 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:22.121801 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:22.122517 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:22.123914 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:22.124319 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:22.124409 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:22.129308 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:22.129325 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:22.129343 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:22.129384 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:22.132021 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:22.133014 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:22.133022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:22.133024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:22.133177 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:22.133551 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:22.133554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:22.133556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:22.133919 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:22.134146 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:22.134502 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 06:43:22.134680 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:43:22.135526 23700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:22.198381 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:22.197836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:46912)
0619 06:43:22.198083 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:22.198200 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:43:22.199601 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:22.199112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:46914)
0619 06:43:22.199328 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:22.199427 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 06:43:22.231817 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 23577
I20260619 06:43:22.244483 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 23217
I20260619 06:43:22.245705 23279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:22.356443 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 23217
I20260619 06:43:42.373891 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34769
--webserver_interface=127.7.146.254
--webserver_port=43739
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:34769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:43:42.690550 23756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:42.690867 23756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:42.691004 23756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:42.698133 23756 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:43:42.698241 23756 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:43:42.698297 23756 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:43:42.698365 23756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:43:42.698441 23756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:42.698534 23756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:42.698594 23756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:43:42.698673 23756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:43:42.708600 23756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:34769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=43739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:42.709908 23756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:42.711220 23756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:42.732496 23762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:42.732528 23763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:42.732553 23765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:42.733198 23756 server_base.cc:1034] running on GCE node
I20260619 06:43:42.734081 23756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:42.735627 23756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:42.736845 23756 hybrid_clock.cc:648] HybridClock initialized: now 1781851422736772 us; error 42 us; skew 500 ppm
I20260619 06:43:42.737210 23756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:42 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851422, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:42.740603 23756 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:43:42.742319 23756 webserver.cc:460] Webserver started at http://127.7.146.254:43739/ using document root <none> and password file <none>
I20260619 06:43:42.743088 23756 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:42.743206 23756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:42.749166 23756 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:43:42.752115 23772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:42.753552 23756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:43:42.753746 23756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "980d5b05b14a47cfb3b410f27a239569"
format_stamp: "Formatted at 2026-06-19 06:43:19 on dist-test-slave-qfk4"
server_key: "573cf799e85c9cbc16015e0c08e57737"
server_key_iv: "bd0307d8b7c690e9897e8ac3e317c6df"
server_key_version: "encryptionkey@0"
I20260619 06:43:42.754769 23756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:42.767593 23756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:42.771854 23756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:42.772126 23756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:42.790146 23756 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:34769
I20260619 06:43:42.790165 23823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:34769 every 8 connection(s)
I20260619 06:43:42.791677 23756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:43:42.795148 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 23756
I20260619 06:43:42.795914 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:35943
--local_ip_for_outbound_sockets=127.7.146.193
--tserver_master_addrs=127.7.146.254:34769
--webserver_port=42887
--webserver_interface=127.7.146.193
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:43:42.798978 23824 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:43:42.802402 23824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Bootstrap starting.
I20260619 06:43:42.821949 23824 log.cc:824] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:42.834071 23824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:43:42.834580 23824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Bootstrap complete.
I20260619 06:43:42.843681 23824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:43:42.844599 23824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980d5b05b14a47cfb3b410f27a239569, State: Initialized, Role: FOLLOWER
I20260619 06:43:42.845247 23824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:43:42.845467 23824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:43:42.845588 23824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:43:42.845755 23824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:43:42.849745 23824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:43:42.850273 23824 leader_election.cc:302] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 980d5b05b14a47cfb3b410f27a239569; no voters:
I20260619 06:43:42.850878 23824 leader_election.cc:288] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:43:42.851078 23828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:43:42.851933 23828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 2 LEADER]: Becoming Leader. State: Replica: 980d5b05b14a47cfb3b410f27a239569, State: Running, Role: LEADER
I20260619 06:43:42.852536 23828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:43:42.852923 23824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:43:42.855175 23830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 980d5b05b14a47cfb3b410f27a239569. Latest consensus state: current_term: 2 leader_uuid: "980d5b05b14a47cfb3b410f27a239569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } } }
I20260619 06:43:42.855556 23830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: This master's current role is: LEADER
I20260619 06:43:42.855947 23829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "980d5b05b14a47cfb3b410f27a239569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } } }
I20260619 06:43:42.856228 23829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: This master's current role is: LEADER
I20260619 06:43:42.866932 23835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:43:42.875973 23835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:43:42.877573 23835 catalog_manager.cc:1259] Loaded cluster ID: d2a32ea2731441d6b3367409b3f048fd
I20260619 06:43:42.877779 23835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:43:42.881901 23835 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:43:42.883611 23835 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Loaded TSK: 0
I20260619 06:43:42.918092 23835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Generated new TSK 1
I20260619 06:43:42.918536 23835 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:43:43.137581 23826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:43.137890 23826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:43.138026 23826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:43.145622 23826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:43.145761 23826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:43.145920 23826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:43:43.156003 23826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:35943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=42887
--tserver_master_addrs=127.7.146.254:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:43.157341 23826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:43.158740 23826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:43.173363 23852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:43.173940 23854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:43.174008 23851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:43.174265 23826 server_base.cc:1034] running on GCE node
I20260619 06:43:43.175012 23826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:43.176030 23826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:43.177286 23826 hybrid_clock.cc:648] HybridClock initialized: now 1781851423177218 us; error 52 us; skew 500 ppm
I20260619 06:43:43.177682 23826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:43 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851423, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:43.181092 23826 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:43:43.182773 23826 webserver.cc:460] Webserver started at http://127.7.146.193:42887/ using document root <none> and password file <none>
I20260619 06:43:43.183493 23826 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:43.183640 23826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:43.189646 23826 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:43:43.192667 23861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:43.194182 23826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:43:43.194363 23826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "08342fc6c0eb48cb98716f9ed8eaf287"
format_stamp: "Formatted at 2026-06-19 06:43:20 on dist-test-slave-qfk4"
server_key: "3e134de4600882f47b1164dbbdfb7d5e"
server_key_iv: "9e9178ac6531e79b692caf5e48c15973"
server_key_version: "encryptionkey@0"
I20260619 06:43:43.195372 23826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:43.213541 23826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:43.217861 23826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:43.218115 23826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:43.219112 23826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:43:43.221050 23868 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 06:43:43.226279 23868 ts_tablet_manager.cc:1852] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 06:43:43.227270 23868 ts_tablet_manager.cc:1905] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:43:43.234627 23868 ts_tablet_manager.cc:1918] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:43:43.234900 23868 ts_tablet_manager.cc:1939] T 70cee97e14d442fbbfd025ea881abaf6 P 08342fc6c0eb48cb98716f9ed8eaf287: Deleting consensus metadata
I20260619 06:43:43.235311 23826 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 06:43:43.235407 23826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.000s
I20260619 06:43:43.235529 23826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:43:43.235599 23826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:43.264823 23826 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:35943
I20260619 06:43:43.264873 23974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:35943 every 8 connection(s)
I20260619 06:43:43.266168 23826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:43:43.268142 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 23826
I20260619 06:43:43.268918 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:39989
--local_ip_for_outbound_sockets=127.7.146.194
--tserver_master_addrs=127.7.146.254:34769
--webserver_port=40263
--webserver_interface=127.7.146.194
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:43:43 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851423, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:43:43.290386 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:43.269170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:59901 (local address 127.7.146.254:34769)
0619 06:43:43.269763 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:43.269774 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:43.270466 (+ 692us) server_negotiation.cc:407] Connection header received
0619 06:43:43.272957 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:43.272971 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:43.273362 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:43.273703 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:43.275281 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:43.276749 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:43.278291 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:43.278942 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:43.279128 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:43.282740 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:43.282749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:43.282764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:43.282807 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:43.285890 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:43.286817 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:43.286824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:43.286830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:43.287006 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:43.287480 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:43.287483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:43.287485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:43.287974 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:43.288148 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:43.289137 (+ 989us) server_negotiation.cc:299] Negotiation successful
0619 06:43:43.289462 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":221,"threads_started":1}
I20260619 06:43:43.290764 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:43.269526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:59901)
0619 06:43:43.270288 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:43.270326 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:43.272365 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:43.273905 (+ 1540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:43.273915 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:43.274532 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:43:43.275087 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:43.275106 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:43.276908 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:43.276912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:43.278153 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:43.278166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:43.279167 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:43.279174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:43.279416 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:43.280028 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:43:43.280063 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:43:43.282411 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:43:43.286045 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:43.286054 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:43.286072 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:43.286686 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:43.287140 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:43.287145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:43.287149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:43.287376 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:43.288126 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:43:43.288134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:43:43.288592 (+ 458us) client_negotiation.cc:769] Sending connection context
0619 06:43:43.288944 (+ 352us) client_negotiation.cc:240] Negotiation successful
0619 06:43:43.289518 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":304,"threads_started":1}
I20260619 06:43:43.293931 23975 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34769
I20260619 06:43:43.294219 23975 heartbeater.cc:464] Registering TS with master...
I20260619 06:43:43.294878 23975 heartbeater.cc:511] Master 127.7.146.254:34769 requested a full tablet report, sending...
I20260619 06:43:43.297113 23789 ts_manager.cc:194] Registered new tserver with Master: 08342fc6c0eb48cb98716f9ed8eaf287 (127.7.146.193:35943)
I20260619 06:43:43.299166 23789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:59901
W20260619 06:43:43.588508 23979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:43.588829 23979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:43.588924 23979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:43.596030 23979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:43.596141 23979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:43.596267 23979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:43:43.606002 23979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:39989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=40263
--tserver_master_addrs=127.7.146.254:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:43.607210 23979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:43.608451 23979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:43.620818 23984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:43.620903 23985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:43.620899 23987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:43.621951 23979 server_base.cc:1034] running on GCE node
I20260619 06:43:43.622538 23979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:43.623487 23979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:43.624696 23979 hybrid_clock.cc:648] HybridClock initialized: now 1781851423624608 us; error 55 us; skew 500 ppm
I20260619 06:43:43.625030 23979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:43 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851423, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:43.628034 23979 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:43:43.629490 23979 webserver.cc:460] Webserver started at http://127.7.146.194:40263/ using document root <none> and password file <none>
I20260619 06:43:43.630167 23979 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:43.630290 23979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:43.635797 23979 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:43:43.638607 23994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:43.639982 23979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:43:43.640138 23979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "cf10a2ea76e34c4d9599fe61f7e7777e"
format_stamp: "Formatted at 2026-06-19 06:43:20 on dist-test-slave-qfk4"
server_key: "41320e298804df891289125d42219d2a"
server_key_iv: "96ade8f2197917ac8b1c389cdb754ec2"
server_key_version: "encryptionkey@0"
I20260619 06:43:43.641167 23979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:43.658776 23979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:43.662649 23979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:43.662904 23979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:43.663878 23979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:43:43.665222 23979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:43:43.665313 23979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:43.665411 23979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:43:43.665480 23979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:43.694761 23979 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:39989
I20260619 06:43:43.694805 24106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:39989 every 8 connection(s)
I20260619 06:43:43.696164 23979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:43:43.700623 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 23979
I20260619 06:43:43.701167 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:43567
--local_ip_for_outbound_sockets=127.7.146.195
--tserver_master_addrs=127.7.146.254:34769
--webserver_port=37443
--webserver_interface=127.7.146.195
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:43:43 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851423, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:43:43.715783 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:43.698697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:60073 (local address 127.7.146.254:34769)
0619 06:43:43.698859 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:43.698863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:43.700141 (+ 1278us) server_negotiation.cc:407] Connection header received
0619 06:43:43.702055 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:43.702060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:43.702152 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:43.702246 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:43.703739 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:43.704441 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:43.706011 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:43.706439 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:43.706549 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:43.709880 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:43.709888 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:43.709892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:43.709932 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:43.712851 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:43.713551 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:43.713557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:43.713560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:43.713701 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:43.714076 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:43.714080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:43.714084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:43.714417 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:43.714561 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:43.715234 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 06:43:43.715466 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:43:43.716658 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:43.699225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:60073)
0619 06:43:43.699976 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:43.700034 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:43.701791 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:43.702495 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:43.702504 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:43.703016 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:43:43.703570 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:43.703587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:43.704613 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:43.704619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:43.705872 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:43.705884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:43.706647 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:43.706654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:43.706881 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:43.707506 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:43:43.707528 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:43:43.709639 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:43:43.712992 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:43.712999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:43.713012 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:43.713432 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:43.713795 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:43.713798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:43.713800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:43.713980 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:43.714539 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:43:43.714546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:43:43.714888 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 06:43:43.715105 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 06:43:43.715498 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":255,"threads_started":1}
I20260619 06:43:43.718202 24107 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34769
I20260619 06:43:43.718556 24107 heartbeater.cc:464] Registering TS with master...
I20260619 06:43:43.719519 24107 heartbeater.cc:511] Master 127.7.146.254:34769 requested a full tablet report, sending...
I20260619 06:43:43.721287 23789 ts_manager.cc:194] Registered new tserver with Master: cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989)
I20260619 06:43:43.722221 23789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:60073
W20260619 06:43:44.021885 24110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:43:44.022176 24110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:43:44.022305 24110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:43:44.029577 24110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:43:44.029706 24110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:43:44.029865 24110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:43:44.039738 24110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:43567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=37443
--tserver_master_addrs=127.7.146.254:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:43:44.041090 24110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:43:44.042402 24110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:43:44.055054 24116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:44.055070 24115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:43:44.055189 24118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:43:44.056310 24110 server_base.cc:1034] running on GCE node
I20260619 06:43:44.056977 24110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:43:44.057977 24110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:43:44.059214 24110 hybrid_clock.cc:648] HybridClock initialized: now 1781851424059143 us; error 56 us; skew 500 ppm
I20260619 06:43:44.059518 24110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:43:44 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851424, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:43:44.062522 24110 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:43:44.064035 24110 webserver.cc:460] Webserver started at http://127.7.146.195:37443/ using document root <none> and password file <none>
I20260619 06:43:44.064756 24110 fs_manager.cc:359] Metadata directory not provided
I20260619 06:43:44.064885 24110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:43:44.070369 24110 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:43:44.073180 24125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:44.074622 24110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:43:44.074806 24110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "d541cd83981940c5b7df8fe745e949d0"
format_stamp: "Formatted at 2026-06-19 06:43:21 on dist-test-slave-qfk4"
server_key: "5b00766cb0d2fa76e7b9692777694854"
server_key_iv: "4826f1a13b7cb4c02285cab39bda7b14"
server_key_version: "encryptionkey@0"
I20260619 06:43:44.075767 24110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:43:44.097534 24110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:43:44.101544 24110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:43:44.101810 24110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:43:44.102782 24110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:43:44.104139 24110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:43:44.104241 24110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:44.104379 24110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:43:44.104455 24110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:43:44.134029 24110 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:43567
I20260619 06:43:44.134074 24237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:43567 every 8 connection(s)
I20260619 06:43:44.135450 24110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:43:44.145110 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 24110
Jun 19 06:43:44 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851424, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:43:44.157518 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.137987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:41203 (local address 127.7.146.254:34769)
0619 06:43:44.138153 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.138157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.139385 (+ 1228us) server_negotiation.cc:407] Connection header received
0619 06:43:44.141307 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.141313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.141398 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.141486 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:44.143023 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.143745 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.145741 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.146423 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.146578 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.150774 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:44.150778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:44.150782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:44.150814 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:44.153858 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.154590 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.154597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.154600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.154753 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.155133 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.155138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.155141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.155495 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:44.155631 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.156943 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.157119 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:43:44.158488 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.138502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:41203)
0619 06:43:44.139209 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.139260 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:44.141024 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:44.141750 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:44.141759 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:44.142279 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:43:44.142849 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.142868 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.143903 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.143907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.145579 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.145595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.146681 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.146689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.147026 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.147881 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:43:44.147919 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:43:44.150573 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:43:44.153981 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:44.153987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:44.154001 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:44.154466 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:44.154846 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:43:44.154850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:43:44.154852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:43:44.155034 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:43:44.156161 (+ 1127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:43:44.156168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:43:44.156503 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 06:43:44.156800 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 06:43:44.157286 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":274,"threads_started":1}
I20260619 06:43:44.159950 24238 heartbeater.cc:347] Connected to a master server at 127.7.146.254:34769
I20260619 06:43:44.160235 24238 heartbeater.cc:464] Registering TS with master...
I20260619 06:43:44.160975 24238 heartbeater.cc:511] Master 127.7.146.254:34769 requested a full tablet report, sending...
I20260619 06:43:44.161279 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.146115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.7.146.254:34769)
0619 06:43:44.146673 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.146683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.147227 (+ 544us) server_negotiation.cc:407] Connection header received
0619 06:43:44.147443 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.147447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.147522 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.147635 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:44.149621 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.150452 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.151876 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.152509 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.152658 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.155656 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:44.155661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:44.155665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:44.155694 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:44.158398 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.159289 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.159294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.159298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.159415 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.159758 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.159763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.159767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.160016 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:44.160213 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.160711 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.160922 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":291,"threads_started":1}
I20260619 06:43:44.162451 23789 ts_manager.cc:194] Registered new tserver with Master: d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
I20260619 06:43:44.163534 23789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:41203
I20260619 06:43:44.163980 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:43:44.171711 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.164933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60012 (local address 127.7.146.254:34769)
0619 06:43:44.165152 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.165158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.165614 (+ 456us) server_negotiation.cc:407] Connection header received
0619 06:43:44.165831 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.165835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.165965 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.166083 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:43:44.166567 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.167298 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.169801 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.170227 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.170337 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.170871 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:43:44.171075 (+ 204us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 06:43:44.171428 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60012: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 06:43:44.171821 24241 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60012: token expired
W20260619 06:43:44.177198 23711 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.146.254:34769)
I20260619 06:43:44.190307 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.177762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.7.146.254:34769)
0619 06:43:44.177942 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.177948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.177992 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:43:44.178198 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.178204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.178282 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.178362 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:44.179861 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.180734 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.182160 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.182570 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.182679 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.185310 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:44.185315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:44.185318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:44.185352 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:44.187691 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.188338 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.188344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.188348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.188482 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.188868 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.188873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.188877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.189206 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:44.189451 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.189755 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.189983 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:43:44.193876 23789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:43:44.201318 23789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:43:44.224074 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.213213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.7.146.194:39989)
0619 06:43:44.215179 (+ 1966us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.215187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.215219 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:43:44.215323 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.215331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.215598 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.216601 (+ 1003us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:44.216889 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.218483 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.221618 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.222892 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.223012 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.223162 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.223612 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.223772 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1810,"thread_start_us":1679,"threads_started":1}
I20260619 06:43:44.224135 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.212920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:39989 (local address 127.0.0.1:55764)
0619 06:43:44.213915 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.213966 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:44.214152 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:44.215844 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:44.215852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:44.215887 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:44.216355 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.216369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.218640 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.218646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.221419 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.221433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.223275 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.223279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.223399 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.223432 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:43:44.223548 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 06:43:44.223749 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":22,"thread_start_us":2406,"threads_started":1}
I20260619 06:43:44.225180 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.213766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.7.146.193:35943)
0619 06:43:44.214786 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.214795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.215508 (+ 713us) server_negotiation.cc:407] Connection header received
0619 06:43:44.215740 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.215747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.216333 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.216554 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:44.217424 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.218956 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.222546 (+ 3590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.223886 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.224004 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.224201 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.224718 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.224875 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":1041,"threads_started":1}
I20260619 06:43:44.226154 24248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.213155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:49096)
0619 06:43:44.215398 (+ 2243us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.215415 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:44.215543 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:44.216815 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:44.216821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:44.216848 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:44.217220 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.217233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.219137 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.219143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.222369 (+ 3226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.222385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.224354 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.224360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.224528 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.224560 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:43:44.224665 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:43:44.224791 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2130,"mutex_wait_us":1,"thread_start_us":777,"threads_started":1}
I20260619 06:43:44.227988 24042 tablet_service.cc:1467] Processing CreateTablet for tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8 (DEFAULT_TABLE table=test-table [id=c356aba096224772802d51da312d5f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:43:44.227941 23910 tablet_service.cc:1467] Processing CreateTablet for tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8 (DEFAULT_TABLE table=test-table [id=c356aba096224772802d51da312d5f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:43:44.229200 24042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:43:44.229238 23910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:43:44.229379 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.215675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:43567 (local address 127.0.0.1:57192)
0619 06:43:44.221212 (+ 5537us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.221233 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:44.221342 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:44.222483 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:44.222488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:44.222512 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:44.222840 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.222852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.225029 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.225034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.226951 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.226966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.228663 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.228670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.228830 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.228863 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:43:44.228930 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 06:43:44.229054 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5419,"thread_start_us":5586,"threads_started":1}
I20260619 06:43:44.230477 24253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.215994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.7.146.195:43567)
0619 06:43:44.221894 (+ 5900us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.221902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.221936 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:43:44.222031 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.222038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.222287 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.222489 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:44.223024 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.224891 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.227091 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.228411 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.229700 (+ 1289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.229926 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.229993 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.230151 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5720,"thread_start_us":5841,"threads_started":1}
I20260619 06:43:44.234334 24173 tablet_service.cc:1467] Processing CreateTablet for tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8 (DEFAULT_TABLE table=test-table [id=c356aba096224772802d51da312d5f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:43:44.235780 24173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:43:44.238618 24254 tablet_bootstrap.cc:492] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Bootstrap starting.
I20260619 06:43:44.242262 24254 tablet_bootstrap.cc:654] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Neither blocks nor log segments found. Creating new log.
I20260619 06:43:44.242430 24255 tablet_bootstrap.cc:492] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e: Bootstrap starting.
I20260619 06:43:44.243669 24254 log.cc:824] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:44.245946 24255 tablet_bootstrap.cc:654] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e: Neither blocks nor log segments found. Creating new log.
I20260619 06:43:44.247120 24255 log.cc:824] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:44.247793 24254 tablet_bootstrap.cc:492] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: No bootstrap required, opened a new log
I20260619 06:43:44.248060 24254 ts_tablet_manager.cc:1397] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 06:43:44.250144 24255 tablet_bootstrap.cc:492] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e: No bootstrap required, opened a new log
I20260619 06:43:44.250468 24255 ts_tablet_manager.cc:1397] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260619 06:43:44.250567 24258 tablet_bootstrap.cc:492] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0: Bootstrap starting.
I20260619 06:43:44.253996 24258 tablet_bootstrap.cc:654] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0: Neither blocks nor log segments found. Creating new log.
I20260619 06:43:44.255190 24258 log.cc:824] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:43:44.258292 24258 tablet_bootstrap.cc:492] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0: No bootstrap required, opened a new log
I20260619 06:43:44.258033 24254 raft_consensus.cc:348] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:43:44.258464 24254 raft_consensus.cc:374] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:43:44.258213 24255 raft_consensus.cc:348] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:43:44.258551 24258 ts_tablet_manager.cc:1397] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260619 06:43:44.258564 24254 raft_consensus.cc:729] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08342fc6c0eb48cb98716f9ed8eaf287, State: Initialized, Role: FOLLOWER
I20260619 06:43:44.258638 24255 raft_consensus.cc:374] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:43:44.258728 24255 raft_consensus.cc:729] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf10a2ea76e34c4d9599fe61f7e7777e, State: Initialized, Role: FOLLOWER
I20260619 06:43:44.259243 24255 consensus_queue.cc:260] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:43:44.259258 24254 consensus_queue.cc:260] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:43:44.260349 24107 heartbeater.cc:503] Master 127.7.146.254:34769 was elected leader, sending a full tablet report...
I20260619 06:43:44.260815 23975 heartbeater.cc:503] Master 127.7.146.254:34769 was elected leader, sending a full tablet report...
I20260619 06:43:44.261349 24254 ts_tablet_manager.cc:1428] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 06:43:44.261998 24255 ts_tablet_manager.cc:1428] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260619 06:43:44.268163 23976 tablet.cc:2369] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:43:44.268801 24258 raft_consensus.cc:348] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:43:44.269213 24258 raft_consensus.cc:374] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:43:44.269308 24258 raft_consensus.cc:729] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d541cd83981940c5b7df8fe745e949d0, State: Initialized, Role: FOLLOWER
I20260619 06:43:44.269338 24260 raft_consensus.cc:482] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:43:44.269570 24260 raft_consensus.cc:504] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:43:44.269862 24258 consensus_queue.cc:260] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:43:44.270882 24238 heartbeater.cc:503] Master 127.7.146.254:34769 was elected leader, sending a full tablet report...
I20260619 06:43:44.271732 24258 ts_tablet_manager.cc:1428] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260619 06:43:44.276824 24260 leader_election.cc:288] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989), d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
I20260619 06:43:44.284166 24262 raft_consensus.cc:482] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:43:44.284461 24262 raft_consensus.cc:504] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:43:44.286602 24262 leader_election.cc:288] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989), 08342fc6c0eb48cb98716f9ed8eaf287 (127.7.146.193:35943)
I20260619 06:43:44.287226 24263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.271595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:39989 (local address 127.7.146.193:33091)
0619 06:43:44.280816 (+ 9221us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.280834 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:44.280976 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:44.281490 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:44.281493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:44.281517 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:44.281835 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.281845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.282925 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.282931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.284764 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.284781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.286447 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.286454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.286625 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.286659 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:43:44.286764 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:43:44.286893 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9101,"thread_start_us":8883,"threads_started":1}
I20260619 06:43:44.287883 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.271577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:33091 (local address 127.7.146.194:39989)
0619 06:43:44.271769 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.271775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.280929 (+ 9154us) server_negotiation.cc:407] Connection header received
0619 06:43:44.281149 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.281155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.281229 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.281295 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:44.281998 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.282774 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.284964 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.286069 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.286230 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.286304 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.287440 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.287579 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:43:44.289681 24062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69ff49d4c874c8dbb3dd1b33c9ccbb8" candidate_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" is_pre_election: true
I20260619 06:43:44.290253 24062 raft_consensus.cc:2454] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08342fc6c0eb48cb98716f9ed8eaf287 in term 0.
I20260619 06:43:44.291031 23864 leader_election.cc:302] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08342fc6c0eb48cb98716f9ed8eaf287, cf10a2ea76e34c4d9599fe61f7e7777e; no voters:
I20260619 06:43:44.293315 24260 raft_consensus.cc:2781] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:43:44.293478 24260 raft_consensus.cc:482] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:43:44.293572 24260 raft_consensus.cc:3037] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:43:44.294893 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.287579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:38733 (local address 127.7.146.194:39989)
0619 06:43:44.288346 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.288352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.288369 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:43:44.288429 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.288434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.288510 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.288635 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:44.289136 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.290216 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.292665 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.293877 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.294032 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.294098 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.294505 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.294624 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"threads_started":1}
I20260619 06:43:44.295174 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.286760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:39989 (local address 127.7.146.195:38733)
0619 06:43:44.287150 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.287167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:44.287282 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:44.288659 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:44.288664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:44.288721 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:44.288963 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.288973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.290375 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.290382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.292477 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.292495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.294039 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.294046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.294332 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.294368 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:43:44.294473 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:43:44.294626 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260619 06:43:44.296272 24260 raft_consensus.cc:504] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:43:44.299494 24264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.281133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:43567 (local address 127.7.146.193:45047)
0619 06:43:44.282258 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.282272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:44.282373 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:44.282637 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:44.282641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:44.282664 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:44.282995 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.283010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.283995 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.284000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.286014 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.286026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.298702 (+ 12676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.298709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.298880 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.298916 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:43:44.299016 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:43:44.299160 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"thread_start_us":891,"threads_started":1}
I20260619 06:43:44.299609 24253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.281191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:45047 (local address 127.7.146.195:43567)
0619 06:43:44.281377 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.281383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.282353 (+ 970us) server_negotiation.cc:407] Connection header received
0619 06:43:44.282483 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.282486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.282542 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.282612 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:44.283141 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.283837 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.286157 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.287293 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.287415 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.287480 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.299062 (+ 11582us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.299240 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:43:44.300396 24260 leader_election.cc:288] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 1 election: Requested vote from peers cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989), d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
I20260619 06:43:44.302249 24062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69ff49d4c874c8dbb3dd1b33c9ccbb8" candidate_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e"
I20260619 06:43:44.302502 24062 raft_consensus.cc:3037] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:43:44.302325 24193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69ff49d4c874c8dbb3dd1b33c9ccbb8" candidate_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d541cd83981940c5b7df8fe745e949d0" is_pre_election: true
I20260619 06:43:44.302891 24193 raft_consensus.cc:2454] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08342fc6c0eb48cb98716f9ed8eaf287 in term 0.
I20260619 06:43:44.303931 24193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69ff49d4c874c8dbb3dd1b33c9ccbb8" candidate_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d541cd83981940c5b7df8fe745e949d0"
I20260619 06:43:44.304160 24193 raft_consensus.cc:3037] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:43:44.306061 24062 raft_consensus.cc:2454] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08342fc6c0eb48cb98716f9ed8eaf287 in term 1.
I20260619 06:43:44.306763 24062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69ff49d4c874c8dbb3dd1b33c9ccbb8" candidate_uuid: "d541cd83981940c5b7df8fe745e949d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" is_pre_election: true
I20260619 06:43:44.307236 24062 raft_consensus.cc:2379] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d541cd83981940c5b7df8fe745e949d0 in current term 1: Already voted for candidate 08342fc6c0eb48cb98716f9ed8eaf287 in this term.
I20260619 06:43:44.307636 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.287078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.7.146.195:50847)
0619 06:43:44.295642 (+ 8564us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.295661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:43:44.295852 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:43:44.296499 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:43:44.296505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:43:44.296639 (+ 134us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:43:44.296902 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.296924 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.298443 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.298448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.300321 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:43:44.300336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.306869 (+ 6533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.306875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:43:44.307085 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.307116 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:43:44.307198 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:43:44.307332 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8457,"thread_start_us":18885,"threads_started":1}
I20260619 06:43:44.307761 24193 raft_consensus.cc:2454] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08342fc6c0eb48cb98716f9ed8eaf287 in term 1.
I20260619 06:43:44.307670 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.287432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:50847 (local address 127.7.146.193:35943)
0619 06:43:44.288178 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.288185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.295745 (+ 7560us) server_negotiation.cc:407] Connection header received
0619 06:43:44.296004 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.296010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.296089 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.296148 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:43:44.297093 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.298261 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.300514 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.301793 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.301917 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.301991 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.307255 (+ 5264us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.307453 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615}
I20260619 06:43:44.308172 23864 leader_election.cc:302] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08342fc6c0eb48cb98716f9ed8eaf287, cf10a2ea76e34c4d9599fe61f7e7777e; no voters:
I20260619 06:43:44.308513 24260 raft_consensus.cc:2781] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:43:44.308737 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69ff49d4c874c8dbb3dd1b33c9ccbb8" candidate_uuid: "d541cd83981940c5b7df8fe745e949d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" is_pre_election: true
I20260619 06:43:44.309091 24260 raft_consensus.cc:686] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 LEADER]: Becoming Leader. State: Replica: 08342fc6c0eb48cb98716f9ed8eaf287, State: Running, Role: LEADER
I20260619 06:43:44.309616 24260 consensus_queue.cc:237] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:43:44.313704 24127 leader_election.cc:302] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d541cd83981940c5b7df8fe745e949d0; no voters: 08342fc6c0eb48cb98716f9ed8eaf287, cf10a2ea76e34c4d9599fe61f7e7777e
I20260619 06:43:44.314118 24262 raft_consensus.cc:2726] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 06:43:44.314363 23787 catalog_manager.cc:5495] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08342fc6c0eb48cb98716f9ed8eaf287 (127.7.146.193). New cstate: current_term: 1 leader_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } health_report { overall_health: UNKNOWN } } }
I20260619 06:43:44.337353 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.332353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.7.146.193:35943)
0619 06:43:44.332509 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.332513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.332550 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:43:44.332799 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.332803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.332883 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.333006 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:43:44.333304 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.334067 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.335405 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.335809 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.335901 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.336322 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:43:44.336891 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:43:44.336997 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.337070 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.337135 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:43:44.345465 24062 raft_consensus.cc:1261] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e [term 1 FOLLOWER]: Refusing update from remote peer 08342fc6c0eb48cb98716f9ed8eaf287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:43:44.345503 24193 raft_consensus.cc:1261] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 1 FOLLOWER]: Refusing update from remote peer 08342fc6c0eb48cb98716f9ed8eaf287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:43:44.346324 24260 consensus_queue.cc:1035] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:43:44.346760 24267 consensus_queue.cc:1035] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:43:44.355285 24269 mvcc.cc:205] Tried to move back new op lower bound from 7298463434108866560 to 7298463433976119296. Current Snapshot: MvccSnapshot[applied={T|T < 7298463434108866560}]
I20260619 06:43:44.357188 24270 mvcc.cc:205] Tried to move back new op lower bound from 7298463434108866560 to 7298463433976119296. Current Snapshot: MvccSnapshot[applied={T|T < 7298463434108866560}]
I20260619 06:43:44.370883 23788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:43:44.371201 23788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:43:44.375074 23788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Sending DeleteTablet for 3 replicas of tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8
I20260619 06:43:44.376238 24042 tablet_service.cc:1514] Processing DeleteTablet for tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:43:44 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:55764
I20260619 06:43:44.376436 24173 tablet_service.cc:1514] Processing DeleteTablet for tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:43:44 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:57192
I20260619 06:43:44.376637 23910 tablet_service.cc:1514] Processing DeleteTablet for tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:43:44 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:49096
I20260619 06:43:44.377499 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 23826
I20260619 06:43:44.389460 24279 tablet_replica.cc:330] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e: stopping tablet replica
I20260619 06:43:44.390122 24279 raft_consensus.cc:2229] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:43:44.390825 24279 raft_consensus.cc:2258] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:43:44.392081 24280 tablet_replica.cc:330] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0: stopping tablet replica
I20260619 06:43:44.392729 24281 tablet_replica.cc:330] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: stopping tablet replica
I20260619 06:43:44.393191 24281 raft_consensus.cc:2229] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:43:44.393667 24280 raft_consensus.cc:2229] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:43:44.394542 24279 ts_tablet_manager.cc:1905] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:43:44.394618 24280 raft_consensus.cc:2258] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:43:44.396309 24281 raft_consensus.cc:2258] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:43:44.397449 24280 ts_tablet_manager.cc:1905] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:43:44.401619 24279 ts_tablet_manager.cc:1918] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:43:44.401800 24279 log.cc:1196] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/b69ff49d4c874c8dbb3dd1b33c9ccbb8
I20260619 06:43:44.402309 24279 ts_tablet_manager.cc:1939] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P cf10a2ea76e34c4d9599fe61f7e7777e: Deleting consensus metadata
I20260619 06:43:44.402798 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.379020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.7.146.193:35943)
0619 06:43:44.379256 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.379264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.379289 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:43:44.379362 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.379368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.379461 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.379599 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:44.382187 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.383443 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.385733 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.386567 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.386694 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.393932 (+ 7238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:44.393951 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:44.393968 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:44.394022 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:44.398373 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.399384 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.399393 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.399397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.399569 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.400050 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.400056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.400059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.400634 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:44.400950 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.402199 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.402437 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:43:44.403584 23970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:44.404186 23775 catalog_manager.cc:4841] TS cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989): tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8 (table test-table [id=c356aba096224772802d51da312d5f77]) successfully deleted
I20260619 06:43:44.404811 24280 ts_tablet_manager.cc:1918] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:43:44.404991 24280 log.cc:1196] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/b69ff49d4c874c8dbb3dd1b33c9ccbb8
I20260619 06:43:44.405532 24280 ts_tablet_manager.cc:1939] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P d541cd83981940c5b7df8fe745e949d0: Deleting consensus metadata
I20260619 06:43:44.407306 23775 catalog_manager.cc:4841] TS d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567): tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8 (table test-table [id=c356aba096224772802d51da312d5f77]) successfully deleted
I20260619 06:43:44.500145 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 23826
W20260619 06:43:44.514832 23774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.146.193:35943 (error 108)
I20260619 06:43:44.514928 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 23979
I20260619 06:43:44.516755 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.516203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:49124)
0619 06:43:44.516414 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.516541 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260619 06:43:44.517009 23774 catalog_manager.cc:4584] TS 08342fc6c0eb48cb98716f9ed8eaf287 (127.7.146.193:35943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b69ff49d4c874c8dbb3dd1b33c9ccbb8: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
I20260619 06:43:44.529302 24266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.515665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.7.146.194:39989)
0619 06:43:44.515859 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.515863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.515885 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:43:44.516108 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.516115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.516198 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.516323 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:44.517982 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.518758 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.520317 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.520917 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.521016 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.523678 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:44.523691 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:44.523706 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:44.523748 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:44.526462 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.527241 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.527251 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.527254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.527426 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.527812 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.527815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.527818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.528162 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:44.528395 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.528793 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.528966 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 06:43:44.529933 24102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:44.570837 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.570257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:49128)
0619 06:43:44.570509 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.570639 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 06:43:44.571941 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.571564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:49136)
0619 06:43:44.571703 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.571792 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 06:43:44.619454 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 23979
I20260619 06:43:44.632673 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.632026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:49144)
0619 06:43:44.632319 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.632430 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 06:43:44.632939 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 24110
I20260619 06:43:44.633723 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.633325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:49146)
0619 06:43:44.633478 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.633568 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:43:44.646173 24253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.633750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.7.146.195:43567)
0619 06:43:44.633990 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:43:44.633996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:43:44.634015 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:43:44.634078 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:43:44.634082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:43:44.634153 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:43:44.634234 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:43:44.635559 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.636310 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.637764 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:43:44.638183 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:43:44.638280 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:43:44.640897 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:43:44.640913 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:43:44.640929 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:43:44.640968 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:43:44.643612 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.644257 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.644263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.644266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.644399 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:43:44.644786 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:43:44.644789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:43:44.644791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:43:44.645156 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:43:44.645370 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 06:43:44.645641 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 06:43:44.645839 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 06:43:44.646881 24233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:44.714491 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.713921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:49154)
0619 06:43:44.714173 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.714303 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 06:43:44.715592 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:43:44.715152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35943 (local address 127.0.0.1:49164)
0619 06:43:44.715367 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 06:43:44.715455 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 06:43:44.736937 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 24110
I20260619 06:43:44.749897 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 23756
I20260619 06:43:44.750983 23819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:43:44.866473 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 23756
I20260619 06:44:04.885416 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34769
--webserver_interface=127.7.146.254
--webserver_port=43739
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:34769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:05.208981 24287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:05.209304 24287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:05.209446 24287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:05.216732 24287 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:44:05.216836 24287 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:44:05.216892 24287 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:44:05.216961 24287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:05.217037 24287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:05.217133 24287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:05.217192 24287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:05.217265 24287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:05.227226 24287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:34769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:34769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=43739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:05.228520 24287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:05.229862 24287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:05.241031 24295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:05.241119 24292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:05.241140 24293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:05.241936 24287 server_base.cc:1034] running on GCE node
I20260619 06:44:05.242683 24287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:05.244058 24287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:05.245308 24287 hybrid_clock.cc:648] HybridClock initialized: now 1781851445245241 us; error 47 us; skew 500 ppm
I20260619 06:44:05.245676 24287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:05 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851445, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:05.253275 24287 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:44:05.254799 24287 webserver.cc:460] Webserver started at http://127.7.146.254:43739/ using document root <none> and password file <none>
I20260619 06:44:05.255494 24287 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:05.255594 24287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:05.260995 24287 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:44:05.263837 24302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:05.265407 24287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:44:05.265568 24287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "980d5b05b14a47cfb3b410f27a239569"
format_stamp: "Formatted at 2026-06-19 06:43:19 on dist-test-slave-qfk4"
server_key: "573cf799e85c9cbc16015e0c08e57737"
server_key_iv: "bd0307d8b7c690e9897e8ac3e317c6df"
server_key_version: "encryptionkey@0"
I20260619 06:44:05.266602 24287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:05.284790 24287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:05.288887 24287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:05.289139 24287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:05.306121 24287 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:34769
I20260619 06:44:05.306147 24353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:34769 every 8 connection(s)
I20260619 06:44:05.307595 24287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:05.308214 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 24287
I20260619 06:44:05.309122 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:35943
--local_ip_for_outbound_sockets=127.7.146.193
--tserver_master_addrs=127.7.146.254:34769
--webserver_port=42887
--webserver_interface=127.7.146.193
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:44:05.314563 24354 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:44:05.317812 24354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Bootstrap starting.
I20260619 06:44:05.337761 24354 log.cc:824] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:05.354341 24354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:44:05.354877 24354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Bootstrap complete.
I20260619 06:44:05.363651 24354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:44:05.364540 24354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980d5b05b14a47cfb3b410f27a239569, State: Initialized, Role: FOLLOWER
I20260619 06:44:05.365177 24354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:44:05.365469 24354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:05.365574 24354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:05.365736 24354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:44:05.369450 24354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:44:05.369946 24354 leader_election.cc:302] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 980d5b05b14a47cfb3b410f27a239569; no voters:
I20260619 06:44:05.370538 24354 leader_election.cc:288] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 06:44:05.370839 24358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:44:05.371774 24358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [term 3 LEADER]: Becoming Leader. State: Replica: 980d5b05b14a47cfb3b410f27a239569, State: Running, Role: LEADER
I20260619 06:44:05.372464 24354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:05.372346 24358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } }
I20260619 06:44:05.376806 24359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "980d5b05b14a47cfb3b410f27a239569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } } }
I20260619 06:44:05.377187 24359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:05.377739 24360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 980d5b05b14a47cfb3b410f27a239569. Latest consensus state: current_term: 3 leader_uuid: "980d5b05b14a47cfb3b410f27a239569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980d5b05b14a47cfb3b410f27a239569" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 34769 } } }
I20260619 06:44:05.378026 24360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:05.388811 24367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:05.397439 24367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:05.398921 24367 catalog_manager.cc:1259] Loaded cluster ID: d2a32ea2731441d6b3367409b3f048fd
I20260619 06:44:05.399032 24367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:05.403229 24367 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:05.404884 24367 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Loaded TSK: 0
I20260619 06:44:05.405025 24367 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 980d5b05b14a47cfb3b410f27a239569: Loaded TSK: 1
F20260619 06:44:05.406112 24367 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f106871c9ad google::LogMessage::Fail() at ??:0
@ 0x7f106871f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f106871c4de google::LogMessage::Flush() at ??:0
@ 0x7f106871da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f106c4807d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f106c46f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f107e14b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f107e14a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f107e1d62a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f107e1d60c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f107e2205e9 std::function<>::operator()() at ??:0
@ 0x7f107e14ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f107e14cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f107e1d4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f107e1d4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f106b368345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f106b36b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f106b36b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f106b346eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f106e6206db start_thread at ??:0
@ 0x7f10675b661f clone at ??:0
W20260619 06:44:05.714587 24356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:05.714874 24356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:05.714969 24356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:05.722294 24356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:05.722445 24356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:05.722575 24356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:44:05.732401 24356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:35943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=42887
--tserver_master_addrs=127.7.146.254:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:05.733703 24356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:05.734952 24356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:05.747823 24402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:05.747936 24401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:05.747893 24404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:05.748843 24356 server_base.cc:1034] running on GCE node
I20260619 06:44:05.749399 24356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:05.750372 24356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:05.751583 24356 hybrid_clock.cc:648] HybridClock initialized: now 1781851445751523 us; error 60 us; skew 500 ppm
I20260619 06:44:05.751876 24356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:05 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851445, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:05.755085 24356 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:44:05.756759 24356 webserver.cc:460] Webserver started at http://127.7.146.193:42887/ using document root <none> and password file <none>
I20260619 06:44:05.757443 24356 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:05.757556 24356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:05.763343 24356 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 06:44:05.766554 24411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:05.767956 24356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:44:05.768122 24356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "08342fc6c0eb48cb98716f9ed8eaf287"
format_stamp: "Formatted at 2026-06-19 06:43:20 on dist-test-slave-qfk4"
server_key: "3e134de4600882f47b1164dbbdfb7d5e"
server_key_iv: "9e9178ac6531e79b692caf5e48c15973"
server_key_version: "encryptionkey@0"
I20260619 06:44:05.769184 24356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:05.789490 24356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:05.793529 24356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:05.793803 24356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:05.794787 24356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:05.796520 24418 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 06:44:05.802253 24356 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 06:44:05.802366 24356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260619 06:44:05.802466 24356 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 06:44:05.805630 24356 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 06:44:05.805708 24356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260619 06:44:05.806154 24418 tablet_bootstrap.cc:492] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Bootstrap starting.
I20260619 06:44:05.814716 24418 log.cc:824] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:05.830348 24418 tablet_bootstrap.cc:492] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:44:05.831282 24418 tablet_bootstrap.cc:492] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Bootstrap complete.
I20260619 06:44:05.831911 24418 ts_tablet_manager.cc:1397] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20260619 06:44:05.840633 24356 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:35943
I20260619 06:44:05.840667 24525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:35943 every 8 connection(s)
I20260619 06:44:05.842017 24356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:44:05.842506 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 24356
I20260619 06:44:05.841674 24418 raft_consensus.cc:348] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:44:05.842916 24418 raft_consensus.cc:729] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08342fc6c0eb48cb98716f9ed8eaf287, State: Initialized, Role: FOLLOWER
I20260619 06:44:05.843302 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:39989
--local_ip_for_outbound_sockets=127.7.146.194
--tserver_master_addrs=127.7.146.254:34769
--webserver_port=40263
--webserver_interface=127.7.146.194
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:44:05.843931 24418 consensus_queue.cc:260] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:44:05.848062 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:05.845570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:47935)
0619 06:44:05.846441 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:05.846748 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":376,"threads_started":1}
W20260619 06:44:05.849308 24526 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:34769 (0 consecutive failures): Network error: Failed to ping master at 127.7.146.254:34769: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
I20260619 06:44:05.850255 24418 ts_tablet_manager.cc:1428] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260619 06:44:05.850337 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:05.849874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:50343)
0619 06:44:05.850114 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:05.850198 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 06:44:05.851517 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:05.851111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:42959)
0619 06:44:05.851274 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:05.851367 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:44:05.851756 24526 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:44:06.164551 24530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:06.164876 24530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:06.165014 24530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:06.175031 24530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:06.175176 24530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:06.175315 24530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:44:06.186158 24530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:39989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=40263
--tserver_master_addrs=127.7.146.254:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:06.187453 24530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:06.188854 24530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:06.201478 24536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:06.201512 24535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:06.202032 24538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:06.202291 24530 server_base.cc:1034] running on GCE node
I20260619 06:44:06.202867 24530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:06.203835 24530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:06.205065 24530 hybrid_clock.cc:648] HybridClock initialized: now 1781851446205019 us; error 32 us; skew 500 ppm
I20260619 06:44:06.205410 24530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:06 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851446, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:06.208596 24530 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:44:06.210067 24530 webserver.cc:460] Webserver started at http://127.7.146.194:40263/ using document root <none> and password file <none>
I20260619 06:44:06.210736 24530 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:06.210855 24530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:06.216156 24530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:44:06.218999 24545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:06.220378 24530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:44:06.220548 24530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "cf10a2ea76e34c4d9599fe61f7e7777e"
format_stamp: "Formatted at 2026-06-19 06:43:20 on dist-test-slave-qfk4"
server_key: "41320e298804df891289125d42219d2a"
server_key_iv: "96ade8f2197917ac8b1c389cdb754ec2"
server_key_version: "encryptionkey@0"
I20260619 06:44:06.221585 24530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:06.234905 24530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:06.238819 24530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:06.239064 24530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:06.240062 24530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:06.241366 24530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:06.241469 24530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:06.241575 24530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:06.241642 24530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:06.271765 24530 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:39989
I20260619 06:44:06.271811 24657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:39989 every 8 connection(s)
I20260619 06:44:06.273288 24530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:44:06.278169 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:06.276227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:57777)
0619 06:44:06.277036 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:06.277289 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":318,"threads_started":1}
I20260619 06:44:06.278525 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 24530
W20260619 06:44:06.278908 24658 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:34769 (0 consecutive failures): Network error: Failed to ping master at 127.7.146.254:34769: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
I20260619 06:44:06.279120 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:43567
--local_ip_for_outbound_sockets=127.7.146.195
--tserver_master_addrs=127.7.146.254:34769
--webserver_port=37443
--webserver_interface=127.7.146.195
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:44:06.279804 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:06.279411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:44037)
0619 06:44:06.279570 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:06.279662 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:44:06.280762 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:06.280380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:60351)
0619 06:44:06.280517 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:06.280625 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:44:06.280987 24658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:44:06.596261 24661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:06.596547 24661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:06.596705 24661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:06.603941 24661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:06.604070 24661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:06.604238 24661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:44:06.613929 24661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:43567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=37443
--tserver_master_addrs=127.7.146.254:34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:06.615161 24661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:06.616465 24661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:06.629334 24666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:06.629283 24667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:06.629303 24669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:06.630263 24661 server_base.cc:1034] running on GCE node
I20260619 06:44:06.630956 24661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:06.631891 24661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:06.633091 24661 hybrid_clock.cc:648] HybridClock initialized: now 1781851446633036 us; error 47 us; skew 500 ppm
I20260619 06:44:06.633409 24661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:06 dist-test-slave-qfk4 krb5kdc[23204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851446, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:06.636363 24661 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:44:06.637749 24661 webserver.cc:460] Webserver started at http://127.7.146.195:37443/ using document root <none> and password file <none>
I20260619 06:44:06.638394 24661 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:06.638489 24661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:06.643987 24661 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:44:06.646837 24676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:06.648234 24661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:44:06.648378 24661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "d541cd83981940c5b7df8fe745e949d0"
format_stamp: "Formatted at 2026-06-19 06:43:21 on dist-test-slave-qfk4"
server_key: "5b00766cb0d2fa76e7b9692777694854"
server_key_iv: "4826f1a13b7cb4c02285cab39bda7b14"
server_key_version: "encryptionkey@0"
I20260619 06:44:06.649391 24661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:06.665683 24661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:06.669803 24661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:06.670022 24661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:06.670912 24661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:06.672276 24661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:06.672359 24661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:06.672439 24661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:06.672493 24661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:06.702865 24661 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:43567
I20260619 06:44:06.702931 24788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:43567 every 8 connection(s)
I20260619 06:44:06.704298 24661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:44:06.709139 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:06.707255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:34309)
0619 06:44:06.708007 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:06.708253 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":272,"threads_started":1}
W20260619 06:44:06.709909 24789 heartbeater.cc:644] Failed to heartbeat to 127.7.146.254:34769 (0 consecutive failures): Network error: Failed to ping master at 127.7.146.254:34769: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
I20260619 06:44:06.710752 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:06.710382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:38691)
0619 06:44:06.710519 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:06.710594 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 06:44:06.711336 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 24661
I20260619 06:44:06.711580 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:06.711261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:36437)
0619 06:44:06.711388 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:06.711473 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 06:44:06.711824 24789 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:44:06.853276 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:06.852400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:38297)
0619 06:44:06.852942 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:06.853055 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":301,"threads_started":1}
I20260619 06:44:07.017227 24794 raft_consensus.cc:482] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:07.017496 24794 raft_consensus.cc:504] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:44:07.019164 24794 leader_election.cc:288] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989), d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
Jun 19 06:44:07 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851445, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
Jun 19 06:44:07 dist-test-slave-qfk4 krb5kdc[23204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851445, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:44:07.042235 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:07.019503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:39989 (local address 127.7.146.193:45729)
0619 06:44:07.019676 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:07.019721 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:07.020325 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:07.023754 (+ 3429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:07.023765 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:07.024532 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:07.025396 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:07.025417 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:07.027235 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:07.027239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:07.028646 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:07.028659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:07.029633 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:07.029640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:07.029862 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:07.030574 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:44:07.030597 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:07.033310 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:07.037622 (+ 4312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:07.037632 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:07.037648 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:07.038408 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:07.038987 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:07.038993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:07.038997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:07.039311 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:07.040345 (+ 1034us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:07.040353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:07.040881 (+ 528us) client_negotiation.cc:769] Sending connection context
0619 06:44:07.041211 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 06:44:07.041782 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":271,"spinlock_wait_cycles":37248}
I20260619 06:44:07.042292 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:07.019914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:43567 (local address 127.7.146.193:48409)
0619 06:44:07.021134 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:07.021152 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:07.021258 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:07.023840 (+ 2582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:07.023845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:07.024615 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:07.025402 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:07.025418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:07.027222 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:07.027225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:07.028643 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:07.028655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:07.029609 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:07.029616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:07.029861 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:07.030621 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:44:07.030638 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:07.033084 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:07.037622 (+ 4538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:07.037632 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:07.037648 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:07.038408 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:07.038987 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:07.038992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:07.038997 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:07.039290 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:07.040345 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:07.040353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:07.040881 (+ 528us) client_negotiation.cc:769] Sending connection context
0619 06:44:07.041211 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 06:44:07.041927 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":332,"spinlock_wait_cycles":38528,"thread_start_us":920,"threads_started":1}
I20260619 06:44:07.042352 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:07.019679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:45729 (local address 127.7.146.194:39989)
0619 06:44:07.021395 (+ 1716us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:07.021406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:07.021437 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:44:07.022862 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:07.022870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:07.023271 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:07.023596 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:07.025597 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:07.027079 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:07.028773 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:07.029402 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:07.029562 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:07.033585 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:07.033594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:07.033609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:07.033653 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:07.037432 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:07.038568 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:07.038577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:07.038586 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:07.038876 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:07.039421 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:07.039426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:07.039430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:07.040169 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:07.040383 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:07.041484 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0619 06:44:07.041937 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1534,"thread_start_us":1379,"threads_started":1}
I20260619 06:44:07.042623 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:07.020395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:48409 (local address 127.7.146.195:43567)
0619 06:44:07.021763 (+ 1368us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:07.021773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:07.021806 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:44:07.023216 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:07.023224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:07.023565 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:07.023862 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:07.025592 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:07.027042 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:07.028776 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:07.029398 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:07.029563 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:07.033375 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:07.033385 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:07.033405 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:07.033456 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:07.037434 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:07.038568 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:07.038577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:07.038586 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:07.038848 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:07.039421 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:07.039426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:07.039430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:07.040169 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:07.040383 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:07.041483 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0619 06:44:07.041935 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1203,"thread_start_us":1042,"threads_started":1}
W20260619 06:44:07.045367 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
W20260619 06:44:07.045626 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
I20260619 06:44:07.045723 24414 leader_election.cc:302] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08342fc6c0eb48cb98716f9ed8eaf287; no voters: cf10a2ea76e34c4d9599fe61f7e7777e, d541cd83981940c5b7df8fe745e949d0
I20260619 06:44:07.046059 24794 raft_consensus.cc:2726] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:44:07.282542 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:07.281579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:45601)
0619 06:44:07.282111 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:07.282279 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":250,"threads_started":1}
I20260619 06:44:07.713100 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:07.712353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:47945)
0619 06:44:07.712796 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:07.712903 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":217,"threads_started":1}
I20260619 06:44:07.855046 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:07.854276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:57963)
0619 06:44:07.854789 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:07.854884 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":225,"threads_started":1}
I20260619 06:44:08.284242 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:08.283504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:35729)
0619 06:44:08.283965 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:08.284064 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260619 06:44:08.714977 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:08.714152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:57825)
0619 06:44:08.714695 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:08.714790 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":289,"threads_started":1}
I20260619 06:44:08.782505 24803 raft_consensus.cc:482] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:08.782692 24803 raft_consensus.cc:504] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:44:08.783259 24803 leader_election.cc:288] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989), d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
W20260619 06:44:08.784510 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
W20260619 06:44:08.784828 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
I20260619 06:44:08.784916 24414 leader_election.cc:302] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08342fc6c0eb48cb98716f9ed8eaf287; no voters: cf10a2ea76e34c4d9599fe61f7e7777e, d541cd83981940c5b7df8fe745e949d0
I20260619 06:44:08.785140 24803 raft_consensus.cc:2726] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:44:08.856822 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:08.855959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:37359)
0619 06:44:08.856489 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:08.856628 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":254,"threads_started":1}
I20260619 06:44:09.285990 24805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:09.285150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:43125)
0619 06:44:09.285721 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:09.285818 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":245,"threads_started":1}
I20260619 06:44:09.716837 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:09.715960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:53185)
0619 06:44:09.716481 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:09.716631 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":263,"threads_started":1}
I20260619 06:44:09.858538 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:09.857821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:52211)
0619 06:44:09.858293 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:09.858381 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260619 06:44:10.287834 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:10.286954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:56715)
0619 06:44:10.287457 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:10.287565 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":228,"threads_started":1}
I20260619 06:44:10.521466 24809 raft_consensus.cc:482] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:10.521642 24809 raft_consensus.cc:504] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:44:10.522341 24809 leader_election.cc:288] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989), d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
W20260619 06:44:10.523324 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
W20260619 06:44:10.523509 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
I20260619 06:44:10.523603 24414 leader_election.cc:302] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08342fc6c0eb48cb98716f9ed8eaf287; no voters: cf10a2ea76e34c4d9599fe61f7e7777e, d541cd83981940c5b7df8fe745e949d0
I20260619 06:44:10.523829 24809 raft_consensus.cc:2726] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:44:10.718572 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:10.717773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:39313)
0619 06:44:10.718262 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:10.718379 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":230,"threads_started":1}
I20260619 06:44:10.860495 24811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:10.859641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:56619)
0619 06:44:10.860176 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:10.860278 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":236,"threads_started":1}
I20260619 06:44:11.289582 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:11.288881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:54927)
0619 06:44:11.289328 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:11.289418 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":215,"threads_started":1}
I20260619 06:44:11.720248 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:11.719522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:40559)
0619 06:44:11.719978 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:11.720067 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":258,"threads_started":1}
I20260619 06:44:11.862155 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:11.861449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:43725)
0619 06:44:11.861899 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:11.861996 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260619 06:44:12.291144 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:12.290412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:42095)
0619 06:44:12.290885 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:12.290969 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":223,"threads_started":1}
I20260619 06:44:12.722074 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:12.721251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:33797)
0619 06:44:12.721801 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:12.721893 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":290,"threads_started":1}
I20260619 06:44:12.863847 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:12.863054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:34877)
0619 06:44:12.863525 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:12.863624 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
I20260619 06:44:13.031353 24818 raft_consensus.cc:482] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:13.031553 24818 raft_consensus.cc:504] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:44:13.032167 24818 leader_election.cc:288] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989), d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
W20260619 06:44:13.033180 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
W20260619 06:44:13.033414 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
I20260619 06:44:13.033561 24414 leader_election.cc:302] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08342fc6c0eb48cb98716f9ed8eaf287; no voters: cf10a2ea76e34c4d9599fe61f7e7777e, d541cd83981940c5b7df8fe745e949d0
I20260619 06:44:13.033802 24818 raft_consensus.cc:2726] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:44:13.293118 24819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:13.292138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:44797)
0619 06:44:13.292830 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:13.292933 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":269,"threads_started":1}
I20260619 06:44:13.723977 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:13.723055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:36053)
0619 06:44:13.723669 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:13.723755 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":289,"threads_started":1}
I20260619 06:44:13.865624 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:13.864892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:60515)
0619 06:44:13.865382 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:13.865471 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":214,"threads_started":1}
I20260619 06:44:14.294656 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:14.293987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:57325)
0619 06:44:14.294390 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:14.294473 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":204,"threads_started":1}
I20260619 06:44:14.725767 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:14.724986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:47389)
0619 06:44:14.725468 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:14.725581 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":227,"threads_started":1}
I20260619 06:44:14.867249 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:14.866474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:47269)
0619 06:44:14.866982 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:14.867064 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":225,"threads_started":1}
I20260619 06:44:15.296406 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:15.295644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:49271)
0619 06:44:15.296148 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:15.296244 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":239,"threads_started":1}
I20260619 06:44:15.727591 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:15.726833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:33535)
0619 06:44:15.727333 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:15.727427 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":223,"threads_started":1}
I20260619 06:44:15.869086 24827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:15.868287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:55915)
0619 06:44:15.868830 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:15.868928 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":273,"threads_started":1}
I20260619 06:44:15.903952 24828 raft_consensus.cc:482] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:15.904176 24828 raft_consensus.cc:504] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:44:15.904857 24828 leader_election.cc:288] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989), d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
W20260619 06:44:15.906044 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
W20260619 06:44:15.906235 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
I20260619 06:44:15.906347 24414 leader_election.cc:302] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08342fc6c0eb48cb98716f9ed8eaf287; no voters: cf10a2ea76e34c4d9599fe61f7e7777e, d541cd83981940c5b7df8fe745e949d0
I20260619 06:44:15.906559 24828 raft_consensus.cc:2726] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:44:16.298094 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:16.297390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:33307)
0619 06:44:16.297827 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:16.297919 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":220,"threads_started":1}
I20260619 06:44:16.729462 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:16.728640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:53901)
0619 06:44:16.729160 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:16.729280 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":283,"threads_started":1}
I20260619 06:44:16.870896 24831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:16.870121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:35001)
0619 06:44:16.870596 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:16.870695 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":243,"threads_started":1}
I20260619 06:44:17.299873 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:17.299065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:33501)
0619 06:44:17.299585 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:17.299680 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":237,"threads_started":1}
I20260619 06:44:17.731127 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:17.730377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:38183)
0619 06:44:17.730881 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:17.730966 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":246,"threads_started":1}
I20260619 06:44:17.872723 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:17.871871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:60855)
0619 06:44:17.872372 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:17.872515 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
I20260619 06:44:18.301580 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:18.300884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:51127)
0619 06:44:18.301332 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:18.301411 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":216,"threads_started":1}
I20260619 06:44:18.402238 24836 raft_consensus.cc:482] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:18.402451 24836 raft_consensus.cc:504] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf10a2ea76e34c4d9599fe61f7e7777e" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 39989 } } peers { permanent_uuid: "08342fc6c0eb48cb98716f9ed8eaf287" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35943 } } peers { permanent_uuid: "d541cd83981940c5b7df8fe745e949d0" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 43567 } }
I20260619 06:44:18.403074 24836 leader_election.cc:288] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989), d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567)
W20260619 06:44:18.404055 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf10a2ea76e34c4d9599fe61f7e7777e (127.7.146.194:39989): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
W20260619 06:44:18.404295 24414 leader_election.cc:341] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d541cd83981940c5b7df8fe745e949d0 (127.7.146.195:43567): Not found: Tablet not found: b69ff49d4c874c8dbb3dd1b33c9ccbb8
I20260619 06:44:18.404388 24414 leader_election.cc:302] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08342fc6c0eb48cb98716f9ed8eaf287; no voters: cf10a2ea76e34c4d9599fe61f7e7777e, d541cd83981940c5b7df8fe745e949d0
I20260619 06:44:18.404731 24836 raft_consensus.cc:2726] T b69ff49d4c874c8dbb3dd1b33c9ccbb8 P 08342fc6c0eb48cb98716f9ed8eaf287 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:44:18.732810 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:18.732054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:51635)
0619 06:44:18.732509 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:18.732626 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
I20260619 06:44:18.874549 24838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:18.873629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:60161)
0619 06:44:18.874197 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:18.874329 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":324,"threads_started":1}
I20260619 06:44:19.303236 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:19.302490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:59541)
0619 06:44:19.302968 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:19.303072 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":214,"threads_started":1}
I20260619 06:44:19.734506 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:19.733747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:59155)
0619 06:44:19.734238 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:19.734345 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":238,"threads_started":1}
I20260619 06:44:19.876130 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:19.875391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:58023)
0619 06:44:19.875842 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:19.875941 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":212,"threads_started":1}
I20260619 06:44:20.305037 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:20.304121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:35359)
0619 06:44:20.304685 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:20.304825 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":262,"threads_started":1}
I20260619 06:44:20.736335 24843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:20.735520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:32903)
0619 06:44:20.736032 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:20.736124 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":255,"threads_started":1}
I20260619 06:44:20.878037 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:20.877320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.193:36069)
0619 06:44:20.877755 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:20.877857 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":223,"threads_started":1}
I20260619 06:44:21.306919 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:21.306161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.194:45859)
0619 06:44:21.306611 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:21.306745 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":224,"threads_started":1}
I20260619 06:44:21.715478 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 24356
I20260619 06:44:21.731220 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:21.716796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.7.146.193:35943)
0619 06:44:21.717353 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:21.717364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:21.717396 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:44:21.717474 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:21.717481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:21.717723 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:21.717891 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:21.719579 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:21.720634 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:21.722034 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:21.722615 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:21.722773 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:21.725562 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:21.725574 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:21.725589 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:21.725629 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:21.728441 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:21.729154 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:21.729162 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:21.729164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:21.729302 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:21.729655 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:21.729659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:21.729661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:21.730042 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:21.730322 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:21.730643 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 06:44:21.730920 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":220,"threads_started":1}
I20260619 06:44:21.732611 24521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:21.738168 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:21.737445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:34769 (local address 127.7.146.195:53637)
0619 06:44:21.737890 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:21.738006 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.254:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":230,"threads_started":1}
I20260619 06:44:21.820638 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 24356
I20260619 06:44:21.834435 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 24530
I20260619 06:44:21.847926 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:21.835167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.7.146.194:39989)
0619 06:44:21.835661 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:21.835666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:21.835683 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:44:21.835742 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:21.835745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:21.835808 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:21.835900 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:21.837330 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:21.838116 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:21.839417 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:21.839849 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:21.839943 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:21.842581 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:21.842586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:21.842590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:21.842624 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:21.845278 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:21.846085 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:21.846088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:21.846091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:21.846214 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:21.846597 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:21.846599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:21.846601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:21.846879 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:21.847076 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:21.847388 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 06:44:21.847569 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":259,"threads_started":1}
I20260619 06:44:21.848613 24653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:21.930749 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 24530
I20260619 06:44:21.944311 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 24661
I20260619 06:44:21.958233 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:21.945170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.7.146.195:43567)
0619 06:44:21.945708 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:21.945713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:21.945729 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:44:21.945787 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:21.945792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:21.945863 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:21.945958 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:21.947320 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:21.948114 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:21.949348 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:21.949773 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:21.949881 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:21.952513 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:21.952518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:21.952522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:21.952556 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:21.955479 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:21.956184 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:21.956188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:21.956190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:21.956336 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:21.956828 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:21.956832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:21.956834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:21.957097 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:21.957365 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:21.957640 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 06:44:21.957864 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":271,"threads_started":1}
I20260619 06:44:21.959115 24784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:22.043149 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 24661
2026-06-19T06:44:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63984 ms)
[----------] 30 tests from SecurityITest (287584 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:22 dist-test-slave-qfk4 krb5kdc[24855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:22 dist-test-slave-qfk4 krb5kdc[24855](info): set up 2 sockets
Jun 19 06:44:22 dist-test-slave-qfk4 krb5kdc[24855](info): commencing operation
krb5kdc: starting...
W20260619 06:44:23.064478 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:23 dist-test-slave-qfk4 krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:44:23.296398 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:37663
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:23.615047 24868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:23.615368 24868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:23.615499 24868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:23.622731 24868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:23.622851 24868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:23.622931 24868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:23.623008 24868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:23.623066 24868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:23.632931 24868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:37663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:23.634189 24868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:23.635473 24868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:44:23.647248 24868 server_base.cc:1034] running on GCE node
W20260619 06:44:23.647188 24873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:23.647161 24874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:23.647379 24876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:23.648334 24868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:23.649751 24868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:23.650966 24868 hybrid_clock.cc:648] HybridClock initialized: now 1781851463650913 us; error 38 us; skew 500 ppm
I20260619 06:44:23.651294 24868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:23 dist-test-slave-qfk4 krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851463, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:23.654290 24868 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:44:23.655683 24868 webserver.cc:460] Webserver started at http://127.7.146.254:45703/ using document root <none> and password file <none>
I20260619 06:44:23.656378 24868 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:23.656489 24868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:23.656803 24868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:23.659740 24868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "8271e131c230409c91b1e9903533ca27"
format_stamp: "Formatted at 2026-06-19 06:44:23 on dist-test-slave-qfk4"
server_key: "9b03a2aecdfbb0f54817b3dd4e733c58"
server_key_iv: "0142051dc8aa5d8f336ea8dcf2a5b2c0"
server_key_version: "encryptionkey@0"
I20260619 06:44:23.660468 24868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "8271e131c230409c91b1e9903533ca27"
format_stamp: "Formatted at 2026-06-19 06:44:23 on dist-test-slave-qfk4"
server_key: "9b03a2aecdfbb0f54817b3dd4e733c58"
server_key_iv: "0142051dc8aa5d8f336ea8dcf2a5b2c0"
server_key_version: "encryptionkey@0"
I20260619 06:44:23.665550 24868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 06:44:23.668980 24883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:23.670370 24868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:44:23.670529 24868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "8271e131c230409c91b1e9903533ca27"
format_stamp: "Formatted at 2026-06-19 06:44:23 on dist-test-slave-qfk4"
server_key: "9b03a2aecdfbb0f54817b3dd4e733c58"
server_key_iv: "0142051dc8aa5d8f336ea8dcf2a5b2c0"
server_key_version: "encryptionkey@0"
I20260619 06:44:23.670711 24868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:23.682237 24868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:23.686560 24868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:23.686827 24868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:23.703945 24868 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:37663
I20260619 06:44:23.703964 24934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:37663 every 8 connection(s)
I20260619 06:44:23.705304 24868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:23.708493 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 24868
I20260619 06:44:23.708740 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:23.709352 24935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:23.709549 7755 external_mini_cluster.cc:1428] Setting key b1298884e7d19adf623d99f764591672
I20260619 06:44:23.723940 24935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27: Bootstrap starting.
Jun 19 06:44:23 dist-test-slave-qfk4 krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:23.727242 24935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:23.728274 24935 log.cc:824] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:23.730999 24935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27: No bootstrap required, opened a new log
I20260619 06:44:23.732414 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:23.711513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.7.146.254:37663)
0619 06:44:23.712270 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:23.712281 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:23.712324 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:44:23.714090 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:23.714112 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:23.714496 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:23.714870 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:23.716700 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:23.718051 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:23.719893 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:23.720709 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:23.720909 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:23.725483 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:23.725491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:23.725505 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:23.725548 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:23.728545 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:23.729260 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:23.729266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:23.729272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:23.729441 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:23.729813 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:23.729817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:23.729819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:23.730311 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:23.730570 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:23.731013 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 06:44:23.731446 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":338,"threads_started":1}
I20260619 06:44:23.738523 24935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e131c230409c91b1e9903533ca27" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37663 } }
I20260619 06:44:23.738931 24935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:23.739054 24935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8271e131c230409c91b1e9903533ca27, State: Initialized, Role: FOLLOWER
I20260619 06:44:23.739573 24935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e131c230409c91b1e9903533ca27" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37663 } }
I20260619 06:44:23.739786 24935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:23.739903 24935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:23.740041 24935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:23.742277 24935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e131c230409c91b1e9903533ca27" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37663 } }
I20260619 06:44:23.742722 24935 leader_election.cc:302] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8271e131c230409c91b1e9903533ca27; no voters:
I20260619 06:44:23.743301 24935 leader_election.cc:288] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:23.743409 24940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:23.744015 24940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [term 1 LEADER]: Becoming Leader. State: Replica: 8271e131c230409c91b1e9903533ca27, State: Running, Role: LEADER
I20260619 06:44:23.744551 24940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e131c230409c91b1e9903533ca27" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37663 } }
I20260619 06:44:23.745443 24935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:23.746976 24941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8271e131c230409c91b1e9903533ca27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e131c230409c91b1e9903533ca27" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37663 } } }
I20260619 06:44:23.747263 24941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:23.747896 24942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8271e131c230409c91b1e9903533ca27. Latest consensus state: current_term: 1 leader_uuid: "8271e131c230409c91b1e9903533ca27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8271e131c230409c91b1e9903533ca27" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37663 } } }
I20260619 06:44:23.748886 24942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:23.749441 24946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:23.755750 24946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:23.765681 24946 catalog_manager.cc:1347] Generated new cluster ID: 02dbd3b630664003b923348637003932
I20260619 06:44:23.765806 24946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:23.777542 24946 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:23.778817 24946 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:23.784809 24946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27: Generated new TSK 0
I20260619 06:44:23.785622 24946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:44:23.852408 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37663
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:24.201896 24961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:24.202164 24961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:24.202281 24961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:24.209472 24961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:24.209580 24961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:24.209714 24961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:44:24.219584 24961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:24.220923 24961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:24.222198 24961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:24.234709 24969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:24.234733 24966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:24.234756 24967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:24.235491 24961 server_base.cc:1034] running on GCE node
I20260619 06:44:24.236020 24961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:24.236963 24961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:24.238154 24961 hybrid_clock.cc:648] HybridClock initialized: now 1781851464238102 us; error 43 us; skew 500 ppm
I20260619 06:44:24.238445 24961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:24 dist-test-slave-qfk4 krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851464, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:24.241501 24961 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:44:24.243108 24961 webserver.cc:460] Webserver started at http://127.7.146.193:41861/ using document root <none> and password file <none>
I20260619 06:44:24.243769 24961 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:24.243899 24961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:24.244190 24961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:24.247198 24961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "9aa439a3157344cda33a19de081b4c95"
format_stamp: "Formatted at 2026-06-19 06:44:24 on dist-test-slave-qfk4"
server_key: "4beceecb3a578e06d374e5685501c68f"
server_key_iv: "dd1c48f6c6eed408e54ce3dcbb4e4895"
server_key_version: "encryptionkey@0"
I20260619 06:44:24.247938 24961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "9aa439a3157344cda33a19de081b4c95"
format_stamp: "Formatted at 2026-06-19 06:44:24 on dist-test-slave-qfk4"
server_key: "4beceecb3a578e06d374e5685501c68f"
server_key_iv: "dd1c48f6c6eed408e54ce3dcbb4e4895"
server_key_version: "encryptionkey@0"
I20260619 06:44:24.252828 24961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 06:44:24.256443 24976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:24.257786 24961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:44:24.257977 24961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "9aa439a3157344cda33a19de081b4c95"
format_stamp: "Formatted at 2026-06-19 06:44:24 on dist-test-slave-qfk4"
server_key: "4beceecb3a578e06d374e5685501c68f"
server_key_iv: "dd1c48f6c6eed408e54ce3dcbb4e4895"
server_key_version: "encryptionkey@0"
I20260619 06:44:24.258167 24961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:24.298077 24961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:24.302103 24961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:24.302389 24961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:24.303339 24961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:24.304674 24961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:24.304749 24961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:24.304862 24961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:24.304911 24961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:24.334811 24961 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:37099
I20260619 06:44:24.334865 25088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:37099 every 8 connection(s)
I20260619 06:44:24.336026 24961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:44:24.339269 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 24961
I20260619 06:44:24.339458 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:44:24.340078 7755 external_mini_cluster.cc:1428] Setting key 61c6c4e1107da42cf95ecf427f2beca5
Jun 19 06:44:24 dist-test-slave-qfk4 krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851464, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:24.356808 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:24.338602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:34597 (local address 127.7.146.254:37663)
0619 06:44:24.339156 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:24.339163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:24.340123 (+ 960us) server_negotiation.cc:407] Connection header received
0619 06:44:24.342212 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:24.342218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:24.342302 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:24.342415 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:24.343994 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:24.345002 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:24.346747 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:24.347323 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:24.347449 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:24.351193 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:24.351198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:24.351201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:24.351238 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:24.353734 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:24.354597 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:24.354601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:24.354604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:24.354723 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:24.355143 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:24.355146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:24.355148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:24.355406 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:24.355524 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:24.356315 (+ 791us) server_negotiation.cc:299] Negotiation successful
0619 06:44:24.356486 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":334,"threads_started":1}
I20260619 06:44:24.358148 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:24.339163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:37663 (local address 127.7.146.193:34597)
0619 06:44:24.339948 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:24.339993 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:24.341953 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:24.342629 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:24.342637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:24.343224 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:24.343816 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:24.343834 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:24.345186 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:24.345191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:24.346608 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:24.346624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:24.347538 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:24.347543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:24.347764 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:24.348410 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:44:24.348432 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:24.350939 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:24.353861 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:24.353870 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:24.353888 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:24.354498 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:24.354830 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:24.354835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:24.354838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:24.355059 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:24.355523 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:24.355532 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:24.355947 (+ 415us) client_negotiation.cc:769] Sending connection context
0619 06:44:24.356258 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 06:44:24.356815 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":307,"threads_started":1}
I20260619 06:44:24.359723 25089 heartbeater.cc:347] Connected to a master server at 127.7.146.254:37663
I20260619 06:44:24.360038 25089 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:24.360846 25089 heartbeater.cc:511] Master 127.7.146.254:37663 requested a full tablet report, sending...
I20260619 06:44:24.363025 24900 ts_manager.cc:194] Registered new tserver with Master: 9aa439a3157344cda33a19de081b4c95 (127.7.146.193:37099)
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
I20260619 06:44:24.364501 24900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:34597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:44:24.393085 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37663
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:24.707145 25095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:24.707412 25095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:24.707531 25095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:24.714797 25095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:24.714913 25095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:24.715072 25095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:44:24.724920 25095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:24.726161 25095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:24.727449 25095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:24.739990 25101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:24.739961 25103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:24.740535 25100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:24.740839 25095 server_base.cc:1034] running on GCE node
I20260619 06:44:24.741392 25095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:24.742326 25095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:24.743526 25095 hybrid_clock.cc:648] HybridClock initialized: now 1781851464743470 us; error 51 us; skew 500 ppm
I20260619 06:44:24.743825 25095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:24 dist-test-slave-qfk4 krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851464, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:24.750473 25095 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:44:24.751992 25095 webserver.cc:460] Webserver started at http://127.7.146.194:45775/ using document root <none> and password file <none>
I20260619 06:44:24.752697 25095 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:24.752808 25095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:24.753103 25095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:24.756222 25095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "535c8beab81248debe78c570df8c13cf"
format_stamp: "Formatted at 2026-06-19 06:44:24 on dist-test-slave-qfk4"
server_key: "f63a470e582b8263cecfef852e36ef20"
server_key_iv: "201da79e95da038bea704c95812f418a"
server_key_version: "encryptionkey@0"
I20260619 06:44:24.757122 25095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "535c8beab81248debe78c570df8c13cf"
format_stamp: "Formatted at 2026-06-19 06:44:24 on dist-test-slave-qfk4"
server_key: "f63a470e582b8263cecfef852e36ef20"
server_key_iv: "201da79e95da038bea704c95812f418a"
server_key_version: "encryptionkey@0"
I20260619 06:44:24.762576 25095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:44:24.766381 25110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:24.767879 25095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:44:24.768033 25095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "535c8beab81248debe78c570df8c13cf"
format_stamp: "Formatted at 2026-06-19 06:44:24 on dist-test-slave-qfk4"
server_key: "f63a470e582b8263cecfef852e36ef20"
server_key_iv: "201da79e95da038bea704c95812f418a"
server_key_version: "encryptionkey@0"
I20260619 06:44:24.768231 25095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:24.788834 25095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:24.793092 25095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:24.793368 25095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:24.794335 25095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:24.795616 25095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:24.795683 25095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:24.795802 25095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:24.795850 25095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:24.825642 25095 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:35433
I20260619 06:44:24.825707 25222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:35433 every 8 connection(s)
I20260619 06:44:24.826900 25095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:44:24.834956 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 25095
I20260619 06:44:24.835124 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:44:24.835579 7755 external_mini_cluster.cc:1428] Setting key dc106d247201a849e4e5c5af041cc50a
Jun 19 06:44:24 dist-test-slave-qfk4 krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851464, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:24.847874 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:24.829547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:33511 (local address 127.7.146.254:37663)
0619 06:44:24.829746 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:24.829752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:24.830972 (+ 1220us) server_negotiation.cc:407] Connection header received
0619 06:44:24.832874 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:24.832880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:24.832954 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:24.833034 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:24.834632 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:24.835429 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:24.837539 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:24.838007 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:24.838127 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:24.841721 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:24.841727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:24.841731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:24.841765 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:24.844423 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:24.845251 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:24.845257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:24.845260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:24.845486 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:24.845993 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:24.845998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:24.846002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:24.846384 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:24.846521 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:24.847315 (+ 794us) server_negotiation.cc:299] Negotiation successful
0619 06:44:24.847553 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:44:24.848788 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:24.829979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:37663 (local address 127.7.146.194:33511)
0619 06:44:24.830793 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:24.830846 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:24.832544 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:24.833230 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:24.833238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:24.833802 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:24.834421 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:24.834441 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:24.835587 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:24.835593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:24.837383 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:24.837398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:24.838242 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:24.838250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:24.838471 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:24.839097 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:44:24.839125 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:24.841476 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:24.844608 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:24.844618 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:24.844636 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:24.845107 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:24.845635 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:24.845640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:24.845643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:24.845866 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:24.846518 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:24.846525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:24.846920 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 06:44:24.847172 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 06:44:24.847610 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":311,"threads_started":1}
I20260619 06:44:24.850274 25223 heartbeater.cc:347] Connected to a master server at 127.7.146.254:37663
I20260619 06:44:24.850616 25223 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:24.851464 25223 heartbeater.cc:511] Master 127.7.146.254:37663 requested a full tablet report, sending...
I20260619 06:44:24.853080 24900 ts_manager.cc:194] Registered new tserver with Master: 535c8beab81248debe78c570df8c13cf (127.7.146.194:35433)
I20260619 06:44:24.853933 24900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:33511
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:44:24.887815 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37663
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:25.207949 25228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:25.208252 25228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:25.208370 25228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:25.215639 25228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:25.215773 25228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:25.215934 25228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:44:25.225802 25228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:25.227058 25228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:25.228350 25228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:25.240895 25234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:25.240926 25233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:25.241576 25236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:25.241585 25228 server_base.cc:1034] running on GCE node
I20260619 06:44:25.242197 25228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:25.243150 25228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:25.244390 25228 hybrid_clock.cc:648] HybridClock initialized: now 1781851465244332 us; error 42 us; skew 500 ppm
I20260619 06:44:25.244791 25228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:25 dist-test-slave-qfk4 krb5kdc[24855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851465, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:25.248132 25228 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:44:25.249606 25228 webserver.cc:460] Webserver started at http://127.7.146.195:44725/ using document root <none> and password file <none>
I20260619 06:44:25.250278 25228 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:25.250394 25228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:25.250666 25228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:25.253785 25228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "86e5e63e26874c13a4a9033b6425e315"
format_stamp: "Formatted at 2026-06-19 06:44:25 on dist-test-slave-qfk4"
server_key: "c6e02889341b4c2d2159c330f2d8e468"
server_key_iv: "a265736ac1cd770895d5548b40f8bfc6"
server_key_version: "encryptionkey@0"
I20260619 06:44:25.254596 25228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "86e5e63e26874c13a4a9033b6425e315"
format_stamp: "Formatted at 2026-06-19 06:44:25 on dist-test-slave-qfk4"
server_key: "c6e02889341b4c2d2159c330f2d8e468"
server_key_iv: "a265736ac1cd770895d5548b40f8bfc6"
server_key_version: "encryptionkey@0"
I20260619 06:44:25.259904 25228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:44:25.263500 25243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:25.264959 25228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:44:25.265098 25228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "86e5e63e26874c13a4a9033b6425e315"
format_stamp: "Formatted at 2026-06-19 06:44:25 on dist-test-slave-qfk4"
server_key: "c6e02889341b4c2d2159c330f2d8e468"
server_key_iv: "a265736ac1cd770895d5548b40f8bfc6"
server_key_version: "encryptionkey@0"
I20260619 06:44:25.265278 25228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:25.282032 25228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:25.286310 25228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:25.286574 25228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:25.287564 25228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:25.289022 25228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:25.289093 25228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:25.289220 25228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:25.289268 25228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:25.318339 25228 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:45729
I20260619 06:44:25.318393 25355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:45729 every 8 connection(s)
I20260619 06:44:25.319605 25228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:44:25.329869 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 25228
I20260619 06:44:25.330070 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:44:25.330511 7755 external_mini_cluster.cc:1428] Setting key ecca02a31e3166070b73e91ad8f2ce42
Jun 19 06:44:25 dist-test-slave-qfk4 krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851465, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:25.339570 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.322083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:39057 (local address 127.7.146.254:37663)
0619 06:44:25.322246 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:25.322251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:25.323353 (+ 1102us) server_negotiation.cc:407] Connection header received
0619 06:44:25.325243 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:25.325248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:25.325325 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:25.325414 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:25.326918 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.327680 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.329233 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.329681 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.329792 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.333429 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:25.333434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:25.333438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:25.333475 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:25.336594 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:25.337286 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:25.337291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:25.337295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:25.337459 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:25.337865 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:25.337870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:25.337874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:25.338224 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:25.338366 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:25.339026 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 06:44:25.339245 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:44:25.340298 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.322500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:37663 (local address 127.7.146.195:39057)
0619 06:44:25.323181 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:25.323219 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:25.324940 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:25.325650 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:25.325658 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:25.326170 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:25.326749 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.326767 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.327834 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.327838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.329098 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.329110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.329894 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.329901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.330149 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.330821 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:44:25.330844 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:25.333225 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:25.336723 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:25.336729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:25.336741 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:25.337155 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:25.337562 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:25.337565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:25.337567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:25.337740 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:25.338354 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:25.338360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:25.338667 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 06:44:25.338885 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 06:44:25.339253 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":263,"threads_started":1}
I20260619 06:44:25.341712 25356 heartbeater.cc:347] Connected to a master server at 127.7.146.254:37663
I20260619 06:44:25.342008 25356 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:25.342702 25356 heartbeater.cc:511] Master 127.7.146.254:37663 requested a full tablet report, sending...
I20260619 06:44:25.344254 24900 ts_manager.cc:194] Registered new tserver with Master: 86e5e63e26874c13a4a9033b6425e315 (127.7.146.195:45729)
I20260619 06:44:25.345225 24900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:39057
I20260619 06:44:25.346222 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:44:25.362540 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.350634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44856 (local address 127.7.146.254:37663)
0619 06:44:25.350796 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:25.350801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:25.351234 (+ 433us) server_negotiation.cc:407] Connection header received
0619 06:44:25.351369 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:25.351372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:25.351436 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:25.351518 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:25.352862 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.353642 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.355002 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.355427 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.355524 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.357855 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:25.357860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:25.357863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:25.357893 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:25.360222 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:25.360874 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:25.360878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:25.360880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:25.361027 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:25.361322 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:25.361325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:25.361327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:25.361643 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:25.361862 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:25.362155 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 06:44:25.362315 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:44:25.367801 25089 heartbeater.cc:503] Master 127.7.146.254:37663 was elected leader, sending a full tablet report...
I20260619 06:44:25.366252 24900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:44:25.373828 24900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:44:25.394083 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.385088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:35433 (local address 127.0.0.1:44220)
0619 06:44:25.386322 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:25.386364 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:25.386590 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:25.387619 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:25.387627 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:25.387658 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:25.388057 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.388068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.389907 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.389912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.391662 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.391674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.393074 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.393080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.393278 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.393329 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 06:44:25.393455 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 06:44:25.393644 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"spinlock_wait_cycles":7552,"thread_start_us":1026,"threads_started":1}
I20260619 06:44:25.394876 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.385296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.7.146.194:35433)
0619 06:44:25.385869 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:25.385879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:25.386762 (+ 883us) server_negotiation.cc:407] Connection header received
0619 06:44:25.386865 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:25.386872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:25.387131 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:25.387315 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:25.388252 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.389765 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.391810 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.392871 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.393018 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.394314 (+ 1296us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:25.394388 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 06:44:25.394558 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260619 06:44:25.395262 25368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.385605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:37099 (local address 127.0.0.1:53496)
0619 06:44:25.386398 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:25.386421 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:25.386590 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:25.387698 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:25.387702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:25.387730 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:25.388219 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.388232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.390343 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.390348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.392464 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.392478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.394488 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.394494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.394646 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.394679 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:44:25.394771 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:44:25.394898 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"spinlock_wait_cycles":3968,"thread_start_us":757,"threads_started":1}
I20260619 06:44:25.395568 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.386209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.7.146.193:37099)
0619 06:44:25.387144 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:25.387152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:25.387177 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:44:25.387263 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:25.387270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:25.387519 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:25.387703 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:25.388420 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.390187 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.392648 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.394013 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.394138 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.394340 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:25.394860 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 06:44:25.395146 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":1219,"threads_started":1}
I20260619 06:44:25.396234 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.386874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:45729 (local address 127.0.0.1:40158)
0619 06:44:25.388776 (+ 1902us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:25.388799 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:25.388914 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:25.389440 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:25.389444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:25.389471 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:25.389789 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.389802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.391723 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.391729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.393578 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.393591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.395542 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.395548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.395714 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.395743 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:44:25.395836 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:44:25.395948 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1795,"thread_start_us":1645,"threads_started":1}
I20260619 06:44:25.397063 25373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.387353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.7.146.195:45729)
0619 06:44:25.388371 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:25.388378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:25.388899 (+ 521us) server_negotiation.cc:407] Connection header received
0619 06:44:25.389044 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:25.389051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:25.389246 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:25.389429 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:25.389977 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.391558 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.393742 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.395071 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.395182 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.395402 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:25.396618 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0619 06:44:25.396779 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":691,"threads_started":1}
I20260619 06:44:25.399001 25158 tablet_service.cc:1467] Processing CreateTablet for tablet 70722f19e71c4932994f0447a7f9b27a (DEFAULT_TABLE table=test-table [id=5eaca57b38cf4880a54858691d45f599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:25.398993 25024 tablet_service.cc:1467] Processing CreateTablet for tablet 70722f19e71c4932994f0447a7f9b27a (DEFAULT_TABLE table=test-table [id=5eaca57b38cf4880a54858691d45f599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:25.400216 25158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70722f19e71c4932994f0447a7f9b27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:25.400357 25024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70722f19e71c4932994f0447a7f9b27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:25.400338 25291 tablet_service.cc:1467] Processing CreateTablet for tablet 70722f19e71c4932994f0447a7f9b27a (DEFAULT_TABLE table=test-table [id=5eaca57b38cf4880a54858691d45f599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:25.401680 25291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70722f19e71c4932994f0447a7f9b27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:25.411320 25374 tablet_bootstrap.cc:492] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: Bootstrap starting.
I20260619 06:44:25.414201 25375 tablet_bootstrap.cc:492] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: Bootstrap starting.
I20260619 06:44:25.414716 25374 tablet_bootstrap.cc:654] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:25.415732 25376 tablet_bootstrap.cc:492] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: Bootstrap starting.
I20260619 06:44:25.416726 25374 log.cc:824] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:25.417865 25375 tablet_bootstrap.cc:654] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:25.418695 25376 tablet_bootstrap.cc:654] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:25.419193 25375 log.cc:824] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:25.419628 25376 log.cc:824] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:25.422160 25376 tablet_bootstrap.cc:492] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: No bootstrap required, opened a new log
I20260619 06:44:25.422438 25376 ts_tablet_manager.cc:1397] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:44:25.424530 25374 tablet_bootstrap.cc:492] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: No bootstrap required, opened a new log
I20260619 06:44:25.424870 25374 ts_tablet_manager.cc:1397] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260619 06:44:25.424965 25375 tablet_bootstrap.cc:492] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: No bootstrap required, opened a new log
I20260619 06:44:25.425233 25375 ts_tablet_manager.cc:1397] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260619 06:44:25.432744 25374 raft_consensus.cc:348] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535c8beab81248debe78c570df8c13cf" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35433 } } peers { permanent_uuid: "86e5e63e26874c13a4a9033b6425e315" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45729 } } peers { permanent_uuid: "9aa439a3157344cda33a19de081b4c95" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37099 } }
I20260619 06:44:25.432767 25376 raft_consensus.cc:348] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535c8beab81248debe78c570df8c13cf" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35433 } } peers { permanent_uuid: "86e5e63e26874c13a4a9033b6425e315" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45729 } } peers { permanent_uuid: "9aa439a3157344cda33a19de081b4c95" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37099 } }
I20260619 06:44:25.433145 25374 raft_consensus.cc:374] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:25.433247 25376 raft_consensus.cc:374] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:25.433252 25374 raft_consensus.cc:729] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 535c8beab81248debe78c570df8c13cf, State: Initialized, Role: FOLLOWER
I20260619 06:44:25.433363 25376 raft_consensus.cc:729] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e5e63e26874c13a4a9033b6425e315, State: Initialized, Role: FOLLOWER
I20260619 06:44:25.433858 25374 consensus_queue.cc:260] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535c8beab81248debe78c570df8c13cf" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35433 } } peers { permanent_uuid: "86e5e63e26874c13a4a9033b6425e315" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45729 } } peers { permanent_uuid: "9aa439a3157344cda33a19de081b4c95" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37099 } }
I20260619 06:44:25.434079 25376 consensus_queue.cc:260] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535c8beab81248debe78c570df8c13cf" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35433 } } peers { permanent_uuid: "86e5e63e26874c13a4a9033b6425e315" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45729 } } peers { permanent_uuid: "9aa439a3157344cda33a19de081b4c95" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37099 } }
I20260619 06:44:25.434926 25223 heartbeater.cc:503] Master 127.7.146.254:37663 was elected leader, sending a full tablet report...
I20260619 06:44:25.435693 25374 ts_tablet_manager.cc:1428] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260619 06:44:25.436525 25375 raft_consensus.cc:348] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535c8beab81248debe78c570df8c13cf" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35433 } } peers { permanent_uuid: "86e5e63e26874c13a4a9033b6425e315" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45729 } } peers { permanent_uuid: "9aa439a3157344cda33a19de081b4c95" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37099 } }
I20260619 06:44:25.436913 25356 heartbeater.cc:503] Master 127.7.146.254:37663 was elected leader, sending a full tablet report...
I20260619 06:44:25.437021 25375 raft_consensus.cc:374] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:25.437119 25375 raft_consensus.cc:729] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa439a3157344cda33a19de081b4c95, State: Initialized, Role: FOLLOWER
I20260619 06:44:25.437703 25375 consensus_queue.cc:260] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535c8beab81248debe78c570df8c13cf" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35433 } } peers { permanent_uuid: "86e5e63e26874c13a4a9033b6425e315" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45729 } } peers { permanent_uuid: "9aa439a3157344cda33a19de081b4c95" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37099 } }
I20260619 06:44:25.439536 25375 ts_tablet_manager.cc:1428] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 06:44:25.439909 25376 ts_tablet_manager.cc:1428] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: Time spent starting tablet: real 0.017s user 0.008s sys 0.005s
I20260619 06:44:25.563457 25381 raft_consensus.cc:482] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:25.563761 25381 raft_consensus.cc:504] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535c8beab81248debe78c570df8c13cf" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35433 } } peers { permanent_uuid: "86e5e63e26874c13a4a9033b6425e315" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45729 } } peers { permanent_uuid: "9aa439a3157344cda33a19de081b4c95" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37099 } }
I20260619 06:44:25.565515 25381 leader_election.cc:288] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 535c8beab81248debe78c570df8c13cf (127.7.146.194:35433), 9aa439a3157344cda33a19de081b4c95 (127.7.146.193:37099)
W20260619 06:44:25.571156 25357 tablet.cc:2369] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:44:25.572693 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.566192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:41551 (local address 127.7.146.193:37099)
0619 06:44:25.566451 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:25.566458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:25.566479 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:44:25.566556 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:25.566561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:25.566633 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:25.566736 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:25.567111 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.568109 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.570520 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.571719 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.571880 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.571956 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:25.572265 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 06:44:25.572393 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 06:44:25.572726 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.565956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:51017 (local address 127.7.146.194:35433)
0619 06:44:25.566152 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:25.566158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:25.566713 (+ 555us) server_negotiation.cc:407] Connection header received
0619 06:44:25.566857 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:25.566862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:25.566939 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:25.567005 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:25.567485 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.568421 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.570695 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.571864 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.572028 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.572076 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:25.572346 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:44:25.572468 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:44:25.572775 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.565786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:37099 (local address 127.7.146.195:41551)
0619 06:44:25.566017 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:25.566037 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:25.566188 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:25.566776 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:25.566781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:25.566805 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:25.566966 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.566973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.568237 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.568241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.570344 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.570359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.571880 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.571886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.572106 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.572148 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 06:44:25.572241 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:44:25.572469 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":27}
I20260619 06:44:25.572836 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.565960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:35433 (local address 127.7.146.195:51017)
0619 06:44:25.566610 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:25.566623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:25.566727 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:25.567039 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:25.567044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:25.567067 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:25.567340 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.567348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.568538 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.568543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.570564 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:25.570574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.572061 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.572067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:25.572214 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.572242 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:44:25.572313 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 06:44:25.572423 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":1,"thread_start_us":415,"threads_started":1}
I20260619 06:44:25.573773 25178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70722f19e71c4932994f0447a7f9b27a" candidate_uuid: "86e5e63e26874c13a4a9033b6425e315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "535c8beab81248debe78c570df8c13cf" is_pre_election: true
I20260619 06:44:25.573755 25044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70722f19e71c4932994f0447a7f9b27a" candidate_uuid: "86e5e63e26874c13a4a9033b6425e315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa439a3157344cda33a19de081b4c95" is_pre_election: true
I20260619 06:44:25.574245 25178 raft_consensus.cc:2454] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86e5e63e26874c13a4a9033b6425e315 in term 0.
I20260619 06:44:25.574242 25044 raft_consensus.cc:2454] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86e5e63e26874c13a4a9033b6425e315 in term 0.
I20260619 06:44:25.574872 25245 leader_election.cc:302] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 535c8beab81248debe78c570df8c13cf, 86e5e63e26874c13a4a9033b6425e315; no voters:
I20260619 06:44:25.575242 25381 raft_consensus.cc:2781] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:44:25.575441 25381 raft_consensus.cc:482] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:44:25.575542 25381 raft_consensus.cc:3037] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:25.577732 25381 raft_consensus.cc:504] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535c8beab81248debe78c570df8c13cf" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35433 } } peers { permanent_uuid: "86e5e63e26874c13a4a9033b6425e315" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45729 } } peers { permanent_uuid: "9aa439a3157344cda33a19de081b4c95" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37099 } }
I20260619 06:44:25.578435 25381 leader_election.cc:288] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [CANDIDATE]: Term 1 election: Requested vote from peers 535c8beab81248debe78c570df8c13cf (127.7.146.194:35433), 9aa439a3157344cda33a19de081b4c95 (127.7.146.193:37099)
W20260619 06:44:25.578735 25224 tablet.cc:2369] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:44:25.578872 25178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70722f19e71c4932994f0447a7f9b27a" candidate_uuid: "86e5e63e26874c13a4a9033b6425e315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "535c8beab81248debe78c570df8c13cf"
I20260619 06:44:25.579066 25178 raft_consensus.cc:3037] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:25.579829 25044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70722f19e71c4932994f0447a7f9b27a" candidate_uuid: "86e5e63e26874c13a4a9033b6425e315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa439a3157344cda33a19de081b4c95"
I20260619 06:44:25.580039 25044 raft_consensus.cc:3037] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:25.582134 25178 raft_consensus.cc:2454] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86e5e63e26874c13a4a9033b6425e315 in term 1.
I20260619 06:44:25.582561 25245 leader_election.cc:302] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 535c8beab81248debe78c570df8c13cf, 86e5e63e26874c13a4a9033b6425e315; no voters:
I20260619 06:44:25.582827 25381 raft_consensus.cc:2781] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:25.582945 25044 raft_consensus.cc:2454] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86e5e63e26874c13a4a9033b6425e315 in term 1.
I20260619 06:44:25.583298 25381 raft_consensus.cc:686] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 1 LEADER]: Becoming Leader. State: Replica: 86e5e63e26874c13a4a9033b6425e315, State: Running, Role: LEADER
I20260619 06:44:25.583771 25381 consensus_queue.cc:237] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535c8beab81248debe78c570df8c13cf" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35433 } } peers { permanent_uuid: "86e5e63e26874c13a4a9033b6425e315" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45729 } } peers { permanent_uuid: "9aa439a3157344cda33a19de081b4c95" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37099 } }
W20260619 06:44:25.588313 25090 tablet.cc:2369] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:44:25.588513 24899 catalog_manager.cc:5495] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86e5e63e26874c13a4a9033b6425e315 (127.7.146.195). New cstate: current_term: 1 leader_uuid: "86e5e63e26874c13a4a9033b6425e315" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535c8beab81248debe78c570df8c13cf" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86e5e63e26874c13a4a9033b6425e315" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9aa439a3157344cda33a19de081b4c95" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37099 } health_report { overall_health: UNKNOWN } } }
I20260619 06:44:25.636309 25373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.630985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.7.146.195:45729)
0619 06:44:25.631217 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:25.631222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:25.631241 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:44:25.631412 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:25.631416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:25.631499 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:25.631639 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:44:25.632127 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.632922 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.634240 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.634646 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.634742 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.635229 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:44:25.635716 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:44:25.635821 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:25.635975 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 06:44:25.636065 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:44:25.644259 25044 raft_consensus.cc:1261] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95 [term 1 FOLLOWER]: Refusing update from remote peer 86e5e63e26874c13a4a9033b6425e315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:25.644260 25178 raft_consensus.cc:1261] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf [term 1 FOLLOWER]: Refusing update from remote peer 86e5e63e26874c13a4a9033b6425e315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:25.645234 25381 consensus_queue.cc:1035] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa439a3157344cda33a19de081b4c95" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:25.645568 25384 consensus_queue.cc:1035] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "535c8beab81248debe78c570df8c13cf" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:25.670258 24899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:25.670521 24899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:25.674115 24899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8271e131c230409c91b1e9903533ca27: Sending DeleteTablet for 3 replicas of tablet 70722f19e71c4932994f0447a7f9b27a
I20260619 06:44:25.675400 25024 tablet_service.cc:1514] Processing DeleteTablet for tablet 70722f19e71c4932994f0447a7f9b27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:25 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:53496
I20260619 06:44:25.675616 25291 tablet_service.cc:1514] Processing DeleteTablet for tablet 70722f19e71c4932994f0447a7f9b27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:25 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:40158
I20260619 06:44:25.676617 25397 tablet_replica.cc:330] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: stopping tablet replica
I20260619 06:44:25.676537 25158 tablet_service.cc:1514] Processing DeleteTablet for tablet 70722f19e71c4932994f0447a7f9b27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:25 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:44220
I20260619 06:44:25.677225 25397 raft_consensus.cc:2229] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:44:25.679078 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 24961
I20260619 06:44:25.687234 25397 raft_consensus.cc:2258] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:25.687808 25396 tablet_replica.cc:330] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: stopping tablet replica
I20260619 06:44:25.688457 25396 raft_consensus.cc:2229] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:25.689086 25396 raft_consensus.cc:2258] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:25.690876 25397 ts_tablet_manager.cc:1905] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:25.690872 25396 ts_tablet_manager.cc:1905] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:25.690937 25398 tablet_replica.cc:330] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: stopping tablet replica
I20260619 06:44:25.692103 25398 raft_consensus.cc:2229] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:25.696022 25398 raft_consensus.cc:2258] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:25.697997 25396 ts_tablet_manager.cc:1918] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:25.698225 25396 log.cc:1196] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/70722f19e71c4932994f0447a7f9b27a
I20260619 06:44:25.698318 25397 ts_tablet_manager.cc:1918] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:25.698511 25397 log.cc:1196] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/70722f19e71c4932994f0447a7f9b27a
I20260619 06:44:25.698937 25396 ts_tablet_manager.cc:1939] T 70722f19e71c4932994f0447a7f9b27a P 9aa439a3157344cda33a19de081b4c95: Deleting consensus metadata
I20260619 06:44:25.699023 25397 ts_tablet_manager.cc:1939] T 70722f19e71c4932994f0447a7f9b27a P 86e5e63e26874c13a4a9033b6425e315: Deleting consensus metadata
I20260619 06:44:25.699656 25398 ts_tablet_manager.cc:1905] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:25.700801 24885 catalog_manager.cc:4841] TS 86e5e63e26874c13a4a9033b6425e315 (127.7.146.195:45729): tablet 70722f19e71c4932994f0447a7f9b27a (table test-table [id=5eaca57b38cf4880a54858691d45f599]) successfully deleted
I20260619 06:44:25.701116 24886 catalog_manager.cc:4841] TS 9aa439a3157344cda33a19de081b4c95 (127.7.146.193:37099): tablet 70722f19e71c4932994f0447a7f9b27a (table test-table [id=5eaca57b38cf4880a54858691d45f599]) successfully deleted
Jun 19 06:44:25 dist-test-slave-qfk4 krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:44:25.709231 25398 ts_tablet_manager.cc:1918] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:25.709467 25398 log.cc:1196] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/70722f19e71c4932994f0447a7f9b27a
I20260619 06:44:25.710043 25398 ts_tablet_manager.cc:1939] T 70722f19e71c4932994f0447a7f9b27a P 535c8beab81248debe78c570df8c13cf: Deleting consensus metadata
I20260619 06:44:25.710981 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.680817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.7.146.193:37099)
0619 06:44:25.681003 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:25.681009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:25.687375 (+ 6366us) server_negotiation.cc:407] Connection header received
0619 06:44:25.687647 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:25.687653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:25.687738 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:25.687854 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:25.692381 (+ 4527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.693571 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.696548 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.697224 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.697358 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.703780 (+ 6422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:25.703795 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:25.703813 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:25.703863 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:25.707462 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:25.708380 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:25.708390 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:25.708394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:25.708605 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:25.709035 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:25.709040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:25.709043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:25.709502 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:25.709780 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:25.710359 (+ 579us) server_negotiation.cc:299] Negotiation successful
0619 06:44:25.710599 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:44:25.711966 24885 catalog_manager.cc:4841] TS 535c8beab81248debe78c570df8c13cf (127.7.146.194:35433): tablet 70722f19e71c4932994f0447a7f9b27a (table test-table [id=5eaca57b38cf4880a54858691d45f599]) successfully deleted
I20260619 06:44:25.712512 25084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:25.807471 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 24961
I20260619 06:44:25.821549 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 25095
Jun 19 06:44:25 dist-test-slave-qfk4 krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:44:25.837746 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.822294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.7.146.194:35433)
0619 06:44:25.822496 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:25.822501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:25.822533 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:44:25.822709 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:25.822713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:25.822777 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:25.822932 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:25.824650 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.825494 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.827160 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.827585 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.827692 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.831785 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:25.831796 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:25.831810 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:25.831849 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:25.834885 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:25.835541 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:25.835548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:25.835551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:25.835673 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:25.836040 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:25.836044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:25.836046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:25.836540 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:25.836845 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:25.837183 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 06:44:25.837394 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 06:44:25.838423 25218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:25.926976 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 25095
I20260619 06:44:25.947026 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 25228
Jun 19 06:44:25 dist-test-slave-qfk4 krb5kdc[24855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:44:25.964630 25373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:25.947847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.7.146.195:45729)
0619 06:44:25.948014 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:25.948018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:25.948036 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:44:25.948180 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:25.948184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:25.948255 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:25.948337 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:25.950091 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.950873 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.952908 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:25.953395 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:25.953524 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:25.958666 (+ 5142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:25.958680 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:25.958696 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:25.958737 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:25.961725 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:25.962438 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:25.962445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:25.962448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:25.962611 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:25.963037 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:25.963040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:25.963043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:25.963419 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:25.963740 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:25.964055 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 06:44:25.964264 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:44:25.965312 25351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:26.086575 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 25228
I20260619 06:44:26.101886 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 24868
I20260619 06:44:26.102990 24930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:26.214530 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 24868
2026-06-19T06:44:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4173 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T06:44:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:26Z Disabled control of system clock
I20260619 06:44:26.282305 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40877
--webserver_interface=127.7.146.254
--webserver_port=0
--builtin_ntp_servers=127.7.146.212:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:40877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:44:26.603875 25407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:26.604156 25407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:26.604235 25407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:26.611500 25407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:26.611608 25407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:26.611645 25407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:26.611701 25407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:26.611748 25407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:26.620862 25407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:40877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:26.622044 25407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:26.623723 25407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:26.635475 25415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:26.635524 25413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:26.635452 25412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:26.636035 25407 server_base.cc:1034] running on GCE node
I20260619 06:44:26.636865 25407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:26.638240 25407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:26.639437 25407 hybrid_clock.cc:648] HybridClock initialized: now 1781851466639392 us; error 41 us; skew 500 ppm
I20260619 06:44:26.639807 25407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:26.642024 25407 webserver.cc:460] Webserver started at http://127.7.146.254:43069/ using document root <none> and password file <none>
I20260619 06:44:26.642673 25407 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:26.642779 25407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:26.643071 25407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:26.645900 25407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "70dd899bbac742d4b932e813e0989617"
format_stamp: "Formatted at 2026-06-19 06:44:26 on dist-test-slave-qfk4"
server_key: "399a439cf5528531cd240df06195f420"
server_key_iv: "d10cd581510ddcaed650f4a2b3284ddd"
server_key_version: "encryptionkey@0"
I20260619 06:44:26.646647 25407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "70dd899bbac742d4b932e813e0989617"
format_stamp: "Formatted at 2026-06-19 06:44:26 on dist-test-slave-qfk4"
server_key: "399a439cf5528531cd240df06195f420"
server_key_iv: "d10cd581510ddcaed650f4a2b3284ddd"
server_key_version: "encryptionkey@0"
I20260619 06:44:26.651556 25407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:44:26.655009 25421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:26.656358 25407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:44:26.656518 25407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "70dd899bbac742d4b932e813e0989617"
format_stamp: "Formatted at 2026-06-19 06:44:26 on dist-test-slave-qfk4"
server_key: "399a439cf5528531cd240df06195f420"
server_key_iv: "d10cd581510ddcaed650f4a2b3284ddd"
server_key_version: "encryptionkey@0"
I20260619 06:44:26.656754 25407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:26.667501 25407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:26.668335 25407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:26.668689 25407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:26.685636 25407 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:40877
I20260619 06:44:26.685662 25472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:40877 every 8 connection(s)
I20260619 06:44:26.686911 25407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:26.690552 25473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:26.694388 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 25407
I20260619 06:44:26.694558 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:26.695086 7755 external_mini_cluster.cc:1428] Setting key 13b069b6df78af1be70e27da4bbfde0a
I20260619 06:44:26.703526 25473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617: Bootstrap starting.
I20260619 06:44:26.707165 25473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:26.708293 25473 log.cc:824] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:26.708463 25476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:26.697049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.7.146.254:40877)
0619 06:44:26.698006 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:26.698018 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:26.698058 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:44:26.699644 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:26.699667 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:26.700159 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:26.700516 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:26.701080 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:26.702364 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:26.704172 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:26.704896 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:26.705032 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:26.706209 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:26.706215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:26.706229 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:26.706755 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:26.706842 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:26.706863 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:26.706951 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:26.707185 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 06:44:26.707485 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":362,"threads_started":1}
I20260619 06:44:26.711077 25473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617: No bootstrap required, opened a new log
I20260619 06:44:26.718374 25473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd899bbac742d4b932e813e0989617" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40877 } }
I20260619 06:44:26.718721 25473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:26.718792 25473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70dd899bbac742d4b932e813e0989617, State: Initialized, Role: FOLLOWER
I20260619 06:44:26.719342 25473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd899bbac742d4b932e813e0989617" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40877 } }
I20260619 06:44:26.719551 25473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:26.719645 25473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:26.719762 25473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:26.721923 25473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd899bbac742d4b932e813e0989617" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40877 } }
I20260619 06:44:26.722384 25473 leader_election.cc:302] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70dd899bbac742d4b932e813e0989617; no voters:
I20260619 06:44:26.722918 25473 leader_election.cc:288] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:26.723055 25478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:26.723702 25478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [term 1 LEADER]: Becoming Leader. State: Replica: 70dd899bbac742d4b932e813e0989617, State: Running, Role: LEADER
I20260619 06:44:26.724251 25478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd899bbac742d4b932e813e0989617" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40877 } }
I20260619 06:44:26.724875 25473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:26.726945 25479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70dd899bbac742d4b932e813e0989617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd899bbac742d4b932e813e0989617" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40877 } } }
I20260619 06:44:26.726977 25480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70dd899bbac742d4b932e813e0989617. Latest consensus state: current_term: 1 leader_uuid: "70dd899bbac742d4b932e813e0989617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd899bbac742d4b932e813e0989617" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40877 } } }
I20260619 06:44:26.727442 25480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:26.727443 25479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:26.728374 25484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:26.733896 25484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:26.743001 25484 catalog_manager.cc:1347] Generated new cluster ID: f25122b631c64d39bbbfee77f695db5e
I20260619 06:44:26.743112 25484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:26.755676 25484 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:26.756946 25484 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:26.769274 25484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617: Generated new TSK 0
I20260619 06:44:26.770303 25484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:26.778643 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40877
--builtin_ntp_servers=127.7.146.212:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:44:27.137539 25497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:27.137836 25497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:27.137969 25497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:27.145481 25497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:27.145594 25497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:27.145740 25497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:44:27.155329 25497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:27.156621 25497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:27.158401 25497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:27.171299 25502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:27.171265 25503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:27.171341 25505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:27.172027 25497 server_base.cc:1034] running on GCE node
I20260619 06:44:27.172696 25497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:27.173697 25497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:27.174947 25497 hybrid_clock.cc:648] HybridClock initialized: now 1781851467174883 us; error 54 us; skew 500 ppm
I20260619 06:44:27.175308 25497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:27.177757 25497 webserver.cc:460] Webserver started at http://127.7.146.193:38739/ using document root <none> and password file <none>
I20260619 06:44:27.178534 25497 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:27.178642 25497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:27.178979 25497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:27.182118 25497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "1581070aa0b64c788f7dd8cf166a2ae9"
format_stamp: "Formatted at 2026-06-19 06:44:27 on dist-test-slave-qfk4"
server_key: "49fdbda2ab1da9fbb03005b475fd7e17"
server_key_iv: "f8bca118e5bf4ed1f743a1b6d36a8cb4"
server_key_version: "encryptionkey@0"
I20260619 06:44:27.182911 25497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "1581070aa0b64c788f7dd8cf166a2ae9"
format_stamp: "Formatted at 2026-06-19 06:44:27 on dist-test-slave-qfk4"
server_key: "49fdbda2ab1da9fbb03005b475fd7e17"
server_key_iv: "f8bca118e5bf4ed1f743a1b6d36a8cb4"
server_key_version: "encryptionkey@0"
I20260619 06:44:27.188184 25497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:44:27.191761 25511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:27.193269 25497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:44:27.193437 25497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "1581070aa0b64c788f7dd8cf166a2ae9"
format_stamp: "Formatted at 2026-06-19 06:44:27 on dist-test-slave-qfk4"
server_key: "49fdbda2ab1da9fbb03005b475fd7e17"
server_key_iv: "f8bca118e5bf4ed1f743a1b6d36a8cb4"
server_key_version: "encryptionkey@0"
I20260619 06:44:27.193647 25497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:27.211294 25497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:27.212169 25497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:27.212456 25497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:27.213584 25497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:27.215305 25497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:27.215410 25497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:27.215517 25497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:27.215593 25497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:27.245882 25497 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:40997
I20260619 06:44:27.245952 25623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:40997 every 8 connection(s)
I20260619 06:44:27.247226 25497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:44:27.255870 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 25497
I20260619 06:44:27.256048 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:44:27.256546 7755 external_mini_cluster.cc:1428] Setting key 63d79788813783d19a1a2f9e5fd7543d
I20260619 06:44:27.259737 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40877
--builtin_ntp_servers=127.7.146.212:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:44:27.259877 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:27.249691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:47749 (local address 127.7.146.254:40877)
0619 06:44:27.250117 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:27.250121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:27.251286 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 06:44:27.253256 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:27.253262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:27.253344 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:27.253450 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:27.254547 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:27.255278 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:27.256999 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:27.257519 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:27.257631 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:27.259102 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:27.259107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:27.259110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:27.259129 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:27.259169 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:27.259182 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:27.259249 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:27.259507 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 06:44:27.259588 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":230,"threads_started":1}
I20260619 06:44:27.261474 25627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:27.250229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40877 (local address 127.7.146.193:47749)
0619 06:44:27.251132 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:27.251173 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:27.252967 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:27.253692 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:27.253702 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:27.253799 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:27.254369 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:27.254387 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:27.255446 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:27.255450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:27.256855 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:27.256867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:27.257737 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:27.257745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:27.257984 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:27.258737 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:44:27.258741 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:27.258799 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:44:27.258905 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:27.259284 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:27.259294 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:27.259303 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:44:27.259496 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 06:44:27.259935 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":252,"threads_started":1}
I20260619 06:44:27.263046 25624 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40877
I20260619 06:44:27.263372 25624 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:27.264263 25624 heartbeater.cc:511] Master 127.7.146.254:40877 requested a full tablet report, sending...
I20260619 06:44:27.266465 25438 ts_manager.cc:194] Registered new tserver with Master: 1581070aa0b64c788f7dd8cf166a2ae9 (127.7.146.193:40997)
I20260619 06:44:27.268209 25438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.193:47749
W20260619 06:44:27.585053 25628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:27.585337 25628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:27.585446 25628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:27.592813 25628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:27.592917 25628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:27.593050 25628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:44:27.602391 25628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:27.603698 25628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:27.605521 25628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:27.618690 25633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:27.618722 25634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:27.619457 25636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:27.619611 25628 server_base.cc:1034] running on GCE node
I20260619 06:44:27.620232 25628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:27.621284 25628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:27.622524 25628 hybrid_clock.cc:648] HybridClock initialized: now 1781851467622460 us; error 52 us; skew 500 ppm
I20260619 06:44:27.622844 25628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:27.625195 25628 webserver.cc:460] Webserver started at http://127.7.146.194:42379/ using document root <none> and password file <none>
I20260619 06:44:27.625872 25628 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:27.625969 25628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:27.626288 25628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:27.629139 25628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "b9a8646496fa49c6936b0215df400e42"
format_stamp: "Formatted at 2026-06-19 06:44:27 on dist-test-slave-qfk4"
server_key: "2811063ea158b99eddff866d94d71801"
server_key_iv: "52bbbd34ac145dbb5bb44d1f1da347ac"
server_key_version: "encryptionkey@0"
I20260619 06:44:27.629879 25628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "b9a8646496fa49c6936b0215df400e42"
format_stamp: "Formatted at 2026-06-19 06:44:27 on dist-test-slave-qfk4"
server_key: "2811063ea158b99eddff866d94d71801"
server_key_iv: "52bbbd34ac145dbb5bb44d1f1da347ac"
server_key_version: "encryptionkey@0"
I20260619 06:44:27.635142 25628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:44:27.638765 25642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:27.640170 25628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:44:27.640322 25628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "b9a8646496fa49c6936b0215df400e42"
format_stamp: "Formatted at 2026-06-19 06:44:27 on dist-test-slave-qfk4"
server_key: "2811063ea158b99eddff866d94d71801"
server_key_iv: "52bbbd34ac145dbb5bb44d1f1da347ac"
server_key_version: "encryptionkey@0"
I20260619 06:44:27.640542 25628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:27.658782 25628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:27.659627 25628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:27.659907 25628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:27.661055 25628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:27.662643 25628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:27.662732 25628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:27.662832 25628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:27.662899 25628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:27.708986 25628 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:35201
I20260619 06:44:27.709008 25754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:35201 every 8 connection(s)
I20260619 06:44:27.710398 25628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:44:27.715083 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 25628
I20260619 06:44:27.715255 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:44:27.715694 7755 external_mini_cluster.cc:1428] Setting key 023b2c148b7293b4f7d5ac47befd322b
I20260619 06:44:27.718559 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40877
--builtin_ntp_servers=127.7.146.212:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:44:27.725224 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:27.713344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:43571 (local address 127.7.146.254:40877)
0619 06:44:27.713552 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:27.713560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:27.715290 (+ 1730us) server_negotiation.cc:407] Connection header received
0619 06:44:27.718062 (+ 2772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:27.718068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:27.718151 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:27.718259 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:27.719570 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:27.720598 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:27.722204 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:27.722742 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:27.722867 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:27.724302 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:27.724308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:27.724313 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:27.724337 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:27.724378 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:27.724393 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:27.724478 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:27.724796 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 06:44:27.724902 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:44:27.726534 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:27.713874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40877 (local address 127.7.146.194:43571)
0619 06:44:27.715082 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:27.715132 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:27.717721 (+ 2589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:27.718524 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:27.718536 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:27.718665 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:27.719397 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:27.719416 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:27.720744 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:27.720748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:27.722065 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:27.722079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:27.722990 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:27.722996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:27.723233 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:27.723961 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:44:27.723964 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:27.724009 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:44:27.724103 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:27.724535 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:27.724543 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:27.724549 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:44:27.724772 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 06:44:27.725118 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":323,"threads_started":1}
I20260619 06:44:27.727964 25755 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40877
I20260619 06:44:27.728262 25755 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:27.729071 25755 heartbeater.cc:511] Master 127.7.146.254:40877 requested a full tablet report, sending...
I20260619 06:44:27.730445 25438 ts_manager.cc:194] Registered new tserver with Master: b9a8646496fa49c6936b0215df400e42 (127.7.146.194:35201)
I20260619 06:44:27.731253 25438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.194:43571
W20260619 06:44:28.048084 25758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:28.048388 25758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:28.048501 25758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:28.056046 25758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:28.056166 25758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:28.056308 25758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:44:28.065979 25758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:28.067353 25758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:28.069247 25758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:28.083343 25763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:28.083264 25764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:28.083312 25766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:28.084203 25758 server_base.cc:1034] running on GCE node
I20260619 06:44:28.084933 25758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:28.085973 25758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:28.087189 25758 hybrid_clock.cc:648] HybridClock initialized: now 1781851468087138 us; error 50 us; skew 500 ppm
I20260619 06:44:28.087531 25758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:28.090158 25758 webserver.cc:460] Webserver started at http://127.7.146.195:32801/ using document root <none> and password file <none>
I20260619 06:44:28.091045 25758 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:28.091166 25758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:28.091549 25758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:28.094755 25758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "1ee88d457e6648838ab2630111f56b84"
format_stamp: "Formatted at 2026-06-19 06:44:28 on dist-test-slave-qfk4"
server_key: "cf1800210e8ca913261eb297110e7627"
server_key_iv: "4ebca991e9ac4fbec997e26da04e4249"
server_key_version: "encryptionkey@0"
I20260619 06:44:28.095672 25758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "1ee88d457e6648838ab2630111f56b84"
format_stamp: "Formatted at 2026-06-19 06:44:28 on dist-test-slave-qfk4"
server_key: "cf1800210e8ca913261eb297110e7627"
server_key_iv: "4ebca991e9ac4fbec997e26da04e4249"
server_key_version: "encryptionkey@0"
I20260619 06:44:28.101394 25758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:44:28.104998 25772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:28.106360 25758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:44:28.106521 25758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "1ee88d457e6648838ab2630111f56b84"
format_stamp: "Formatted at 2026-06-19 06:44:28 on dist-test-slave-qfk4"
server_key: "cf1800210e8ca913261eb297110e7627"
server_key_iv: "4ebca991e9ac4fbec997e26da04e4249"
server_key_version: "encryptionkey@0"
I20260619 06:44:28.106730 25758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:28.123622 25758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:28.124552 25758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:28.124897 25758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:28.125968 25758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:28.127511 25758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:28.127609 25758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:28.127722 25758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:28.127794 25758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:28.158507 25758 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:34719
I20260619 06:44:28.158567 25884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:34719 every 8 connection(s)
I20260619 06:44:28.159857 25758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:44:28.162711 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 25758
I20260619 06:44:28.162860 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:44:28.163300 7755 external_mini_cluster.cc:1428] Setting key e5322a0b24a683390c3498bd3b245c0d
I20260619 06:44:28.175019 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.162480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:48047 (local address 127.7.146.254:40877)
0619 06:44:28.162694 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:28.162701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:28.164369 (+ 1668us) server_negotiation.cc:407] Connection header received
0619 06:44:28.166771 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:28.166777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:28.166870 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:28.166980 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:28.168264 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.169445 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.171540 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.172186 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.172313 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.174124 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:28.174131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:28.174135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:28.174159 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:28.174201 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:28.174217 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:28.174281 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:28.174576 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 06:44:28.174680 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:44:28.176211 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.163101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40877 (local address 127.7.146.195:48047)
0619 06:44:28.164202 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:28.164259 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:28.166468 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:28.167239 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:28.167252 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:28.167380 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:28.168102 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.168121 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.169605 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.169611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.171402 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.171415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.172440 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.172449 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.172889 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.173692 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:44:28.173696 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:28.173755 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:44:28.173877 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:28.174362 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:28.174370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:28.174376 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:44:28.174541 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 06:44:28.174977 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":331,"threads_started":1}
I20260619 06:44:28.177974 25885 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40877
I20260619 06:44:28.178300 25885 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:28.179141 25885 heartbeater.cc:511] Master 127.7.146.254:40877 requested a full tablet report, sending...
I20260619 06:44:28.180514 25438 ts_manager.cc:194] Registered new tserver with Master: 1ee88d457e6648838ab2630111f56b84 (127.7.146.195:34719)
I20260619 06:44:28.181375 25438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.195:48047
I20260619 06:44:28.191352 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:44:28.201622 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.195555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.7.146.254:40877)
0619 06:44:28.195721 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:28.195725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:28.196079 (+ 354us) server_negotiation.cc:407] Connection header received
0619 06:44:28.196297 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:28.196301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:28.196378 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:28.196466 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:28.197084 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.197833 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.199345 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.199805 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.199898 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.200993 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:28.200997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:28.201001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:28.201020 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:28.201054 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:28.201065 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:28.201134 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:28.201313 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 06:44:28.201394 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:44:28.205562 25438 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:44:28.212531 25438 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:44:28.246009 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.225809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:34719 (local address 127.0.0.1:43970)
0619 06:44:28.237812 (+ 12003us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:28.237827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:28.237941 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:28.238775 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:28.238780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:28.238804 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:28.239186 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.239221 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.240964 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.240970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.243041 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.243053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.244916 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.244929 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.245264 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.245334 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 06:44:28.245466 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 06:44:28.245655 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11869,"mutex_wait_us":58,"thread_start_us":11746,"threads_started":1}
I20260619 06:44:28.246022 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.226460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.7.146.195:34719)
0619 06:44:28.236733 (+ 10273us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:28.237385 (+ 652us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:28.237926 (+ 541us) server_negotiation.cc:407] Connection header received
0619 06:44:28.238070 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:28.238077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:28.238332 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:28.238523 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:28.239423 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.240833 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.243218 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.244668 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.244811 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.245106 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:28.245515 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 06:44:28.245679 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10079,"thread_start_us":10737,"threads_started":1}
I20260619 06:44:28.247629 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.238257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:35201 (local address 127.0.0.1:58142)
0619 06:44:28.240250 (+ 1993us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:28.240266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:28.240377 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:28.241202 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:28.241206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:28.241234 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:28.241685 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.241697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.243486 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.243492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.244969 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.244981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.246876 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.246883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.247044 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.247075 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:44:28.247155 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:44:28.247275 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1878,"mutex_wait_us":3,"thread_start_us":1750,"threads_started":1}
I20260619 06:44:28.247655 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.237052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.7.146.194:35201)
0619 06:44:28.239934 (+ 2882us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:28.239945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:28.240367 (+ 422us) server_negotiation.cc:407] Connection header received
0619 06:44:28.240531 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:28.240538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:28.240800 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:28.240986 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:28.241896 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.243366 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.245249 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.246700 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.246819 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.247056 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:28.247202 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 06:44:28.247358 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2743,"thread_start_us":2656,"threads_started":1}
I20260619 06:44:28.250406 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.225218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:40997 (local address 127.0.0.1:43976)
0619 06:44:28.236253 (+ 11035us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:28.236300 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:28.236622 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:28.238153 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:28.238160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:28.238203 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:28.238615 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.238630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.240591 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.240596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.243098 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.243113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.249611 (+ 6498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.249618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.249806 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.249837 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:44:28.249919 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:44:28.250055 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10789,"mutex_wait_us":26,"thread_start_us":10632,"threads_started":1}
I20260619 06:44:28.251191 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.225496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.7.146.193:40997)
0619 06:44:28.235441 (+ 9945us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:28.235452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:28.236421 (+ 969us) server_negotiation.cc:407] Connection header received
0619 06:44:28.236880 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:28.236888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:28.237168 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:28.237387 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:28.238836 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.240437 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.243271 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.244737 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.244857 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.249493 (+ 4636us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:28.250618 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0619 06:44:28.250817 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9702,"thread_start_us":9953,"threads_started":1}
I20260619 06:44:28.251628 25690 tablet_service.cc:1467] Processing CreateTablet for tablet 2da9c0da7c904a96a6679df313febac4 (DEFAULT_TABLE table=test-table [id=53b7b4e35c7044f6bf30a5cf4bdb4a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:28.252053 25820 tablet_service.cc:1467] Processing CreateTablet for tablet 2da9c0da7c904a96a6679df313febac4 (DEFAULT_TABLE table=test-table [id=53b7b4e35c7044f6bf30a5cf4bdb4a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:28.253268 25690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da9c0da7c904a96a6679df313febac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:28.253422 25820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da9c0da7c904a96a6679df313febac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:28.256009 25559 tablet_service.cc:1467] Processing CreateTablet for tablet 2da9c0da7c904a96a6679df313febac4 (DEFAULT_TABLE table=test-table [id=53b7b4e35c7044f6bf30a5cf4bdb4a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:28.257129 25559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da9c0da7c904a96a6679df313febac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:28.268354 25903 tablet_bootstrap.cc:492] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9: Bootstrap starting.
I20260619 06:44:28.271454 25624 heartbeater.cc:503] Master 127.7.146.254:40877 was elected leader, sending a full tablet report...
I20260619 06:44:28.271739 25903 tablet_bootstrap.cc:654] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:28.273132 25903 log.cc:824] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:28.274520 25905 tablet_bootstrap.cc:492] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: Bootstrap starting.
I20260619 06:44:28.278079 25904 tablet_bootstrap.cc:492] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: Bootstrap starting.
I20260619 06:44:28.279803 25903 tablet_bootstrap.cc:492] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9: No bootstrap required, opened a new log
I20260619 06:44:28.280100 25903 ts_tablet_manager.cc:1397] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260619 06:44:28.281771 25904 tablet_bootstrap.cc:654] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:28.283062 25904 log.cc:824] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:28.283455 25905 tablet_bootstrap.cc:654] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:28.284673 25905 log.cc:824] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:28.286345 25904 tablet_bootstrap.cc:492] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: No bootstrap required, opened a new log
I20260619 06:44:28.286628 25904 ts_tablet_manager.cc:1397] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 06:44:28.287297 25905 tablet_bootstrap.cc:492] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: No bootstrap required, opened a new log
I20260619 06:44:28.287556 25905 ts_tablet_manager.cc:1397] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260619 06:44:28.290114 25903 raft_consensus.cc:348] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 40997 } } peers { permanent_uuid: "b9a8646496fa49c6936b0215df400e42" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35201 } } peers { permanent_uuid: "1ee88d457e6648838ab2630111f56b84" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 34719 } }
I20260619 06:44:28.290613 25903 raft_consensus.cc:374] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:28.290772 25903 raft_consensus.cc:729] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1581070aa0b64c788f7dd8cf166a2ae9, State: Initialized, Role: FOLLOWER
I20260619 06:44:28.291435 25903 consensus_queue.cc:260] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 40997 } } peers { permanent_uuid: "b9a8646496fa49c6936b0215df400e42" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35201 } } peers { permanent_uuid: "1ee88d457e6648838ab2630111f56b84" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 34719 } }
I20260619 06:44:28.294188 25903 ts_tablet_manager.cc:1428] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260619 06:44:28.296195 25905 raft_consensus.cc:348] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 40997 } } peers { permanent_uuid: "b9a8646496fa49c6936b0215df400e42" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35201 } } peers { permanent_uuid: "1ee88d457e6648838ab2630111f56b84" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 34719 } }
I20260619 06:44:28.296325 25904 raft_consensus.cc:348] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 40997 } } peers { permanent_uuid: "b9a8646496fa49c6936b0215df400e42" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35201 } } peers { permanent_uuid: "1ee88d457e6648838ab2630111f56b84" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 34719 } }
I20260619 06:44:28.296711 25905 raft_consensus.cc:374] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:28.296721 25904 raft_consensus.cc:374] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:28.296813 25904 raft_consensus.cc:729] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a8646496fa49c6936b0215df400e42, State: Initialized, Role: FOLLOWER
I20260619 06:44:28.296813 25905 raft_consensus.cc:729] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee88d457e6648838ab2630111f56b84, State: Initialized, Role: FOLLOWER
I20260619 06:44:28.297416 25905 consensus_queue.cc:260] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 40997 } } peers { permanent_uuid: "b9a8646496fa49c6936b0215df400e42" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35201 } } peers { permanent_uuid: "1ee88d457e6648838ab2630111f56b84" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 34719 } }
I20260619 06:44:28.297430 25904 consensus_queue.cc:260] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 40997 } } peers { permanent_uuid: "b9a8646496fa49c6936b0215df400e42" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35201 } } peers { permanent_uuid: "1ee88d457e6648838ab2630111f56b84" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 34719 } }
I20260619 06:44:28.298435 25885 heartbeater.cc:503] Master 127.7.146.254:40877 was elected leader, sending a full tablet report...
I20260619 06:44:28.298972 25905 ts_tablet_manager.cc:1428] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 06:44:28.299039 25904 ts_tablet_manager.cc:1428] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 06:44:28.299616 25755 heartbeater.cc:503] Master 127.7.146.254:40877 was elected leader, sending a full tablet report...
W20260619 06:44:28.411525 25886 tablet.cc:2369] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:44:28.417888 25911 raft_consensus.cc:482] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:28.418171 25911 raft_consensus.cc:504] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 40997 } } peers { permanent_uuid: "b9a8646496fa49c6936b0215df400e42" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35201 } } peers { permanent_uuid: "1ee88d457e6648838ab2630111f56b84" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 34719 } }
I20260619 06:44:28.419818 25911 leader_election.cc:288] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1581070aa0b64c788f7dd8cf166a2ae9 (127.7.146.193:40997), 1ee88d457e6648838ab2630111f56b84 (127.7.146.195:34719)
I20260619 06:44:28.426616 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.420386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:51783 (local address 127.7.146.195:34719)
0619 06:44:28.420646 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:28.420653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:28.421184 (+ 531us) server_negotiation.cc:407] Connection header received
0619 06:44:28.421376 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:28.421380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:28.421438 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:28.421510 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:28.422075 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.422769 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.424777 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.425624 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.425808 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.425878 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:28.426224 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 06:44:28.426354 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:44:28.426644 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.420185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:56875 (local address 127.7.146.193:40997)
0619 06:44:28.420414 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:28.420422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:28.421166 (+ 744us) server_negotiation.cc:407] Connection header received
0619 06:44:28.421371 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:28.421375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:28.421454 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:28.421529 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:28.422119 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.422786 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.424698 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.425577 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.425735 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.425810 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:28.426230 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 06:44:28.426364 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:44:28.426765 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.420270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:34719 (local address 127.7.146.194:51783)
0619 06:44:28.421082 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:28.421097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:28.421248 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:28.421534 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:28.421538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:28.421561 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:28.421931 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.421940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.422891 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.422896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.424599 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.424613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.425812 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.425818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.426060 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.426084 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:44:28.426164 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:44:28.426390 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":133,"spinlock_wait_cycles":896,"thread_start_us":523,"threads_started":1}
I20260619 06:44:28.426824 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.420019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:40997 (local address 127.7.146.194:56875)
0619 06:44:28.421052 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:28.421072 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:28.421240 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:28.421558 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:28.421563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:28.421585 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:28.421982 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.421991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.422914 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.422919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.424493 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:28.424506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.425763 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.425769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:28.426045 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.426082 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:44:28.426223 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:44:28.426424 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":152,"spinlock_wait_cycles":34816,"thread_start_us":621,"threads_started":1}
I20260619 06:44:28.427825 25840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da9c0da7c904a96a6679df313febac4" candidate_uuid: "b9a8646496fa49c6936b0215df400e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee88d457e6648838ab2630111f56b84" is_pre_election: true
I20260619 06:44:28.427824 25579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da9c0da7c904a96a6679df313febac4" candidate_uuid: "b9a8646496fa49c6936b0215df400e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" is_pre_election: true
I20260619 06:44:28.428283 25840 raft_consensus.cc:2454] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9a8646496fa49c6936b0215df400e42 in term 0.
I20260619 06:44:28.428292 25579 raft_consensus.cc:2454] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9a8646496fa49c6936b0215df400e42 in term 0.
I20260619 06:44:28.428926 25644 leader_election.cc:302] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1581070aa0b64c788f7dd8cf166a2ae9, b9a8646496fa49c6936b0215df400e42; no voters:
I20260619 06:44:28.429294 25911 raft_consensus.cc:2781] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:44:28.429446 25911 raft_consensus.cc:482] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:44:28.429545 25911 raft_consensus.cc:3037] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:28.432132 25911 raft_consensus.cc:504] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 40997 } } peers { permanent_uuid: "b9a8646496fa49c6936b0215df400e42" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35201 } } peers { permanent_uuid: "1ee88d457e6648838ab2630111f56b84" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 34719 } }
I20260619 06:44:28.432811 25911 leader_election.cc:288] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [CANDIDATE]: Term 1 election: Requested vote from peers 1581070aa0b64c788f7dd8cf166a2ae9 (127.7.146.193:40997), 1ee88d457e6648838ab2630111f56b84 (127.7.146.195:34719)
I20260619 06:44:28.433255 25579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da9c0da7c904a96a6679df313febac4" candidate_uuid: "b9a8646496fa49c6936b0215df400e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1581070aa0b64c788f7dd8cf166a2ae9"
I20260619 06:44:28.433321 25840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da9c0da7c904a96a6679df313febac4" candidate_uuid: "b9a8646496fa49c6936b0215df400e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee88d457e6648838ab2630111f56b84"
I20260619 06:44:28.433444 25579 raft_consensus.cc:3037] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:28.433528 25840 raft_consensus.cc:3037] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:28.435613 25579 raft_consensus.cc:2454] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9a8646496fa49c6936b0215df400e42 in term 1.
I20260619 06:44:28.435925 25840 raft_consensus.cc:2454] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9a8646496fa49c6936b0215df400e42 in term 1.
I20260619 06:44:28.436008 25644 leader_election.cc:302] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1581070aa0b64c788f7dd8cf166a2ae9, b9a8646496fa49c6936b0215df400e42; no voters:
I20260619 06:44:28.436479 25911 raft_consensus.cc:2781] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:28.436939 25911 raft_consensus.cc:686] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 1 LEADER]: Becoming Leader. State: Replica: b9a8646496fa49c6936b0215df400e42, State: Running, Role: LEADER
I20260619 06:44:28.437458 25911 consensus_queue.cc:237] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 40997 } } peers { permanent_uuid: "b9a8646496fa49c6936b0215df400e42" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35201 } } peers { permanent_uuid: "1ee88d457e6648838ab2630111f56b84" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 34719 } }
I20260619 06:44:28.441992 25438 catalog_manager.cc:5495] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9a8646496fa49c6936b0215df400e42 (127.7.146.194). New cstate: current_term: 1 leader_uuid: "b9a8646496fa49c6936b0215df400e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 40997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9a8646496fa49c6936b0215df400e42" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ee88d457e6648838ab2630111f56b84" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 34719 } health_report { overall_health: UNKNOWN } } }
W20260619 06:44:28.462489 25756 tablet.cc:2369] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:44:28.473145 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.468101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58156 (local address 127.7.146.194:35201)
0619 06:44:28.468301 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:28.468306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:28.468323 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:44:28.468408 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:28.468412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:28.468494 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:28.468624 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:44:28.468920 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.469611 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.470965 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.471464 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.471579 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.472043 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:44:28.472554 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:44:28.472736 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:28.472786 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 06:44:28.472859 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:44:28.481575 25840 raft_consensus.cc:1261] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84 [term 1 FOLLOWER]: Refusing update from remote peer b9a8646496fa49c6936b0215df400e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:28.481588 25579 raft_consensus.cc:1261] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9 [term 1 FOLLOWER]: Refusing update from remote peer b9a8646496fa49c6936b0215df400e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:28.482604 25914 consensus_queue.cc:1035] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ee88d457e6648838ab2630111f56b84" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:28.482939 25911 consensus_queue.cc:1035] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1581070aa0b64c788f7dd8cf166a2ae9" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 40997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:28.506290 25438 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:28.506561 25438 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:28.510051 25438 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70dd899bbac742d4b932e813e0989617: Sending DeleteTablet for 3 replicas of tablet 2da9c0da7c904a96a6679df313febac4
I20260619 06:44:28.511225 25690 tablet_service.cc:1514] Processing DeleteTablet for tablet 2da9c0da7c904a96a6679df313febac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:28 UTC) from {username='slave'} at 127.0.0.1:58142
I20260619 06:44:28.511257 25820 tablet_service.cc:1514] Processing DeleteTablet for tablet 2da9c0da7c904a96a6679df313febac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:28 UTC) from {username='slave'} at 127.0.0.1:43970
I20260619 06:44:28.511257 25559 tablet_service.cc:1514] Processing DeleteTablet for tablet 2da9c0da7c904a96a6679df313febac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:28 UTC) from {username='slave'} at 127.0.0.1:43976
I20260619 06:44:28.512068 25926 tablet_replica.cc:330] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: stopping tablet replica
I20260619 06:44:28.512538 25926 raft_consensus.cc:2229] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:44:28.513273 25926 raft_consensus.cc:2258] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:28.513624 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 25497
I20260619 06:44:28.515416 25926 ts_tablet_manager.cc:1905] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:28.521075 25928 tablet_replica.cc:330] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9: stopping tablet replica
I20260619 06:44:28.521576 25928 raft_consensus.cc:2229] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:28.522147 25928 raft_consensus.cc:2258] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:28.522956 25926 ts_tablet_manager.cc:1918] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:28.523187 25926 log.cc:1196] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/2da9c0da7c904a96a6679df313febac4
I20260619 06:44:28.523828 25926 ts_tablet_manager.cc:1939] T 2da9c0da7c904a96a6679df313febac4 P b9a8646496fa49c6936b0215df400e42: Deleting consensus metadata
I20260619 06:44:28.524668 25928 ts_tablet_manager.cc:1905] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:28.524968 25927 tablet_replica.cc:330] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: stopping tablet replica
I20260619 06:44:28.525539 25927 raft_consensus.cc:2229] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:28.525810 25423 catalog_manager.cc:4841] TS b9a8646496fa49c6936b0215df400e42 (127.7.146.194:35201): tablet 2da9c0da7c904a96a6679df313febac4 (table test-table [id=53b7b4e35c7044f6bf30a5cf4bdb4a17]) successfully deleted
I20260619 06:44:28.526118 25927 raft_consensus.cc:2258] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:28.528100 25927 ts_tablet_manager.cc:1905] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:28.532519 25928 ts_tablet_manager.cc:1918] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:28.532766 25928 log.cc:1196] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/2da9c0da7c904a96a6679df313febac4
I20260619 06:44:28.533277 25928 ts_tablet_manager.cc:1939] T 2da9c0da7c904a96a6679df313febac4 P 1581070aa0b64c788f7dd8cf166a2ae9: Deleting consensus metadata
I20260619 06:44:28.535454 25423 catalog_manager.cc:4841] TS 1581070aa0b64c788f7dd8cf166a2ae9 (127.7.146.193:40997): tablet 2da9c0da7c904a96a6679df313febac4 (table test-table [id=53b7b4e35c7044f6bf30a5cf4bdb4a17]) successfully deleted
I20260619 06:44:28.536521 25927 ts_tablet_manager.cc:1918] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:28.536791 25927 log.cc:1196] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/2da9c0da7c904a96a6679df313febac4
I20260619 06:44:28.537346 25927 ts_tablet_manager.cc:1939] T 2da9c0da7c904a96a6679df313febac4 P 1ee88d457e6648838ab2630111f56b84: Deleting consensus metadata
I20260619 06:44:28.537396 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.515154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.7.146.193:40997)
0619 06:44:28.515358 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:28.515364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:28.529648 (+ 14284us) server_negotiation.cc:407] Connection header received
0619 06:44:28.529862 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:28.529867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:28.529970 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:28.530063 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:28.530484 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.531682 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.533649 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.534248 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.534344 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.535735 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:28.535746 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:28.535764 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:28.536599 (+ 835us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:28.536662 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:28.536688 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:28.536837 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:28.536944 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 06:44:28.537053 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:44:28.538321 25619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:28.539636 25424 catalog_manager.cc:4841] TS 1ee88d457e6648838ab2630111f56b84 (127.7.146.195:34719): tablet 2da9c0da7c904a96a6679df313febac4 (table test-table [id=53b7b4e35c7044f6bf30a5cf4bdb4a17]) successfully deleted
I20260619 06:44:28.625792 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 25497
I20260619 06:44:28.640291 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 25628
I20260619 06:44:28.647770 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.641063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.7.146.194:35201)
0619 06:44:28.641242 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:28.641246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:28.641509 (+ 263us) server_negotiation.cc:407] Connection header received
0619 06:44:28.641701 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:28.641707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:28.641808 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:28.641902 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:28.642316 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.643115 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.644416 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.645011 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.645129 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.646320 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:28.646330 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:28.646344 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:28.646992 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:28.647046 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:28.647068 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:28.647171 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:28.647319 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 06:44:28.647428 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:44:28.648521 25750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:28.760170 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 25628
I20260619 06:44:28.775527 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 25758
I20260619 06:44:28.782167 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:28.776332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.7.146.195:34719)
0619 06:44:28.776513 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:28.776518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:28.776605 (+ 87us) server_negotiation.cc:407] Connection header received
0619 06:44:28.776788 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:28.776792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:28.776877 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:28.776973 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:28.777321 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.778087 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.779418 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:28.779797 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:28.779882 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:28.780918 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:28.780927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:28.780960 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:28.781514 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:28.781561 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:28.781581 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:28.781665 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:28.781753 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 06:44:28.781847 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:44:28.782804 25880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:28.870069 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 25758
I20260619 06:44:28.882712 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 25407
I20260619 06:44:28.883782 25468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:28.991967 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 25407
2026-06-19T06:44:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:29 dist-test-slave-qfk4 krb5kdc[25936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:29 dist-test-slave-qfk4 krb5kdc[25936](info): set up 2 sockets
Jun 19 06:44:29 dist-test-slave-qfk4 krb5kdc[25936](info): commencing operation
krb5kdc: starting...
W20260619 06:44:30.019291 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:30 dist-test-slave-qfk4 krb5kdc[25936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:44:30.259500 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:37709
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:37709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:30.585027 25949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:30.585326 25949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:30.585424 25949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:30.592756 25949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:30.592885 25949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:30.592929 25949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:30.592984 25949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:30.593031 25949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:30.602939 25949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:37709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:37709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:30.604197 25949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:30.605453 25949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:30.616700 25954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:30.616681 25955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:30.616648 25957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:30.617409 25949 server_base.cc:1034] running on GCE node
I20260619 06:44:30.618160 25949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:30.619552 25949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:30.620746 25949 hybrid_clock.cc:648] HybridClock initialized: now 1781851470620700 us; error 30 us; skew 500 ppm
I20260619 06:44:30.621155 25949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:30 dist-test-slave-qfk4 krb5kdc[25936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851470, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:30.627444 25949 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:44:30.628907 25949 webserver.cc:460] Webserver started at http://127.7.146.254:41779/ using document root <none> and password file <none>
I20260619 06:44:30.629594 25949 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:30.629712 25949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:30.630010 25949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:30.632964 25949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "918cf0e6087043178178fb194c7b223a"
format_stamp: "Formatted at 2026-06-19 06:44:30 on dist-test-slave-qfk4"
server_key: "d777e6349d067d2b350a5dd9b45bf688"
server_key_iv: "e0e4cd7665eaba3d4a099604eebee41c"
server_key_version: "encryptionkey@0"
I20260619 06:44:30.633668 25949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "918cf0e6087043178178fb194c7b223a"
format_stamp: "Formatted at 2026-06-19 06:44:30 on dist-test-slave-qfk4"
server_key: "d777e6349d067d2b350a5dd9b45bf688"
server_key_iv: "e0e4cd7665eaba3d4a099604eebee41c"
server_key_version: "encryptionkey@0"
I20260619 06:44:30.638729 25949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:44:30.642185 25964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:30.643533 25949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:44:30.643697 25949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "918cf0e6087043178178fb194c7b223a"
format_stamp: "Formatted at 2026-06-19 06:44:30 on dist-test-slave-qfk4"
server_key: "d777e6349d067d2b350a5dd9b45bf688"
server_key_iv: "e0e4cd7665eaba3d4a099604eebee41c"
server_key_version: "encryptionkey@0"
I20260619 06:44:30.643880 25949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:30.660089 25949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:30.664399 25949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:30.664738 25949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:30.681566 25949 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:37709
I20260619 06:44:30.681588 26015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:37709 every 8 connection(s)
I20260619 06:44:30.682933 25949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:30.686625 26016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:30.692512 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 25949
I20260619 06:44:30.692711 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:30.693156 7755 external_mini_cluster.cc:1428] Setting key fd5dcc1eb72c57011f2077f39e71dca2
I20260619 06:44:30.697408 26016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a: Bootstrap starting.
I20260619 06:44:30.701033 26016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:30.702139 26016 log.cc:824] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:30.705231 26016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a: No bootstrap required, opened a new log
Jun 19 06:44:30 dist-test-slave-qfk4 krb5kdc[25936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:30.715211 26016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918cf0e6087043178178fb194c7b223a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37709 } }
I20260619 06:44:30.715698 26016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:30.715812 26016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918cf0e6087043178178fb194c7b223a, State: Initialized, Role: FOLLOWER
I20260619 06:44:30.716478 26016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918cf0e6087043178178fb194c7b223a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37709 } }
I20260619 06:44:30.716782 26016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:30.716920 26016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:30.717103 26016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:30.717172 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:30.694982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.7.146.254:37709)
0619 06:44:30.695779 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:30.695790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:30.695830 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:44:30.697631 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:30.697654 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:30.698066 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:30.698392 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:30.700213 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:30.701674 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:30.703359 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:30.704039 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:30.704176 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:30.708757 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:30.708768 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:30.708789 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:30.708837 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:30.712696 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:30.713393 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:30.713406 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:30.713415 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:30.713642 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:30.713986 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:30.713990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:30.713993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:30.714722 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:30.715013 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:30.715325 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 06:44:30.715783 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":376,"threads_started":1}
I20260619 06:44:30.719467 26016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918cf0e6087043178178fb194c7b223a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37709 } }
I20260619 06:44:30.719936 26016 leader_election.cc:302] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 918cf0e6087043178178fb194c7b223a; no voters:
I20260619 06:44:30.720450 26016 leader_election.cc:288] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:30.720592 26021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:30.721428 26021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [term 1 LEADER]: Becoming Leader. State: Replica: 918cf0e6087043178178fb194c7b223a, State: Running, Role: LEADER
I20260619 06:44:30.721953 26021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918cf0e6087043178178fb194c7b223a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37709 } }
I20260619 06:44:30.722599 26016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:30.724480 26023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 918cf0e6087043178178fb194c7b223a. Latest consensus state: current_term: 1 leader_uuid: "918cf0e6087043178178fb194c7b223a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918cf0e6087043178178fb194c7b223a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37709 } } }
I20260619 06:44:30.724889 26023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:30.725215 26022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "918cf0e6087043178178fb194c7b223a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918cf0e6087043178178fb194c7b223a" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 37709 } } }
I20260619 06:44:30.725622 26022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:30.730038 26028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:30.734572 26028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:30.743026 26028 catalog_manager.cc:1347] Generated new cluster ID: 9e3b78fa819547889c0d3761ebf22547
I20260619 06:44:30.743139 26028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:30.765479 26028 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:30.767619 26028 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:30.786538 26028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a: Generated new TSK 0
I20260619 06:44:30.787688 26028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:44:30.851399 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37709
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:31.181609 26042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:31.181910 26042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:31.182032 26042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:31.189216 26042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:31.189348 26042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:31.189514 26042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:44:31.199151 26042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:31.200484 26042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:31.201958 26042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:31.215057 26048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:31.215124 26047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:31.215094 26050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:31.215955 26042 server_base.cc:1034] running on GCE node
I20260619 06:44:31.216542 26042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:31.217542 26042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:31.218740 26042 hybrid_clock.cc:648] HybridClock initialized: now 1781851471218675 us; error 61 us; skew 500 ppm
I20260619 06:44:31.219050 26042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:31 dist-test-slave-qfk4 krb5kdc[25936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851471, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:31.222106 26042 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:44:31.223613 26042 webserver.cc:460] Webserver started at http://127.7.146.193:44741/ using document root <none> and password file <none>
I20260619 06:44:31.224313 26042 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:31.224427 26042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:31.224754 26042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:31.227756 26042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "a061745ca9da4bad879ca1f3bf8791c6"
format_stamp: "Formatted at 2026-06-19 06:44:31 on dist-test-slave-qfk4"
server_key: "d4f4066c449f1e42ca388d9c2f4fd452"
server_key_iv: "0d8019a167438646bf29d6d41299661f"
server_key_version: "encryptionkey@0"
I20260619 06:44:31.228555 26042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "a061745ca9da4bad879ca1f3bf8791c6"
format_stamp: "Formatted at 2026-06-19 06:44:31 on dist-test-slave-qfk4"
server_key: "d4f4066c449f1e42ca388d9c2f4fd452"
server_key_iv: "0d8019a167438646bf29d6d41299661f"
server_key_version: "encryptionkey@0"
I20260619 06:44:31.233606 26042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:44:31.237071 26057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:31.238523 26042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:44:31.238677 26042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "a061745ca9da4bad879ca1f3bf8791c6"
format_stamp: "Formatted at 2026-06-19 06:44:31 on dist-test-slave-qfk4"
server_key: "d4f4066c449f1e42ca388d9c2f4fd452"
server_key_iv: "0d8019a167438646bf29d6d41299661f"
server_key_version: "encryptionkey@0"
I20260619 06:44:31.238883 26042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:31.259753 26042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:31.263820 26042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:31.264104 26042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:31.265152 26042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:31.266476 26042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:31.266556 26042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:31.266675 26042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:31.266728 26042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:31.298204 26042 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:44727
I20260619 06:44:31.298242 26169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:44727 every 8 connection(s)
I20260619 06:44:31.299589 26042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:44:31.307214 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 26042
I20260619 06:44:31.307381 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:44:31.307792 7755 external_mini_cluster.cc:1428] Setting key fede2c466eb53468e012a7b60565fe78
Jun 19 06:44:31 dist-test-slave-qfk4 krb5kdc[25936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851471, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:31.319725 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:31.302130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:50379 (local address 127.7.146.254:37709)
0619 06:44:31.302586 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:31.302591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:31.303545 (+ 954us) server_negotiation.cc:407] Connection header received
0619 06:44:31.305418 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:31.305423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:31.305511 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:31.305603 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:31.307102 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:31.307921 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:31.309948 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:31.310327 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:31.310440 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:31.314063 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:31.314068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:31.314072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:31.314104 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:31.316539 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:31.317330 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:31.317336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:31.317340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:31.317501 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:31.317953 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:31.317959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:31.317962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:31.318287 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:31.318420 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:31.319137 (+ 717us) server_negotiation.cc:299] Negotiation successful
0619 06:44:31.319359 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":204,"threads_started":1}
I20260619 06:44:31.320539 26173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:31.302682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:37709 (local address 127.7.146.193:50379)
0619 06:44:31.303385 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:31.303423 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:31.305155 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:31.305851 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:31.305860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:31.306369 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:31.306924 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:31.306941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:31.308058 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:31.308062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:31.309818 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:31.309830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:31.310538 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:31.310545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:31.310794 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:31.311403 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:44:31.311430 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:31.313828 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:31.316702 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:31.316709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:31.316722 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:31.317207 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:31.317612 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:31.317616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:31.317618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:31.317802 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:31.318407 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:31.318413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:31.318733 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 06:44:31.319001 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 06:44:31.319404 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":294,"threads_started":1}
I20260619 06:44:31.322108 26170 heartbeater.cc:347] Connected to a master server at 127.7.146.254:37709
I20260619 06:44:31.322419 26170 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:31.323141 26170 heartbeater.cc:511] Master 127.7.146.254:37709 requested a full tablet report, sending...
I20260619 06:44:31.325241 25981 ts_manager.cc:194] Registered new tserver with Master: a061745ca9da4bad879ca1f3bf8791c6 (127.7.146.193:44727)
I20260619 06:44:31.326795 25981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:50379
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:44:31.359587 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37709
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:31.673980 26176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:31.674286 26176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:31.674413 26176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:31.681988 26176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:31.682121 26176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:31.682276 26176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:44:31.692358 26176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:31.693682 26176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:31.695040 26176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:31.708367 26182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:31.708442 26184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:31.708397 26181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:31.708937 26176 server_base.cc:1034] running on GCE node
I20260619 06:44:31.709650 26176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:31.710733 26176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:31.712006 26176 hybrid_clock.cc:648] HybridClock initialized: now 1781851471711942 us; error 55 us; skew 500 ppm
I20260619 06:44:31.712378 26176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:31 dist-test-slave-qfk4 krb5kdc[25936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851471, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:31.715590 26176 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:44:31.717238 26176 webserver.cc:460] Webserver started at http://127.7.146.194:44257/ using document root <none> and password file <none>
I20260619 06:44:31.717936 26176 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:31.718057 26176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:31.718354 26176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:31.721475 26176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "4175df1c02154afaaa69f1d488cca890"
format_stamp: "Formatted at 2026-06-19 06:44:31 on dist-test-slave-qfk4"
server_key: "c14faf8de3fc28e458c1ecd392cf45bc"
server_key_iv: "07a5c93d8c584bb47d09122b215991ab"
server_key_version: "encryptionkey@0"
I20260619 06:44:31.722236 26176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "4175df1c02154afaaa69f1d488cca890"
format_stamp: "Formatted at 2026-06-19 06:44:31 on dist-test-slave-qfk4"
server_key: "c14faf8de3fc28e458c1ecd392cf45bc"
server_key_iv: "07a5c93d8c584bb47d09122b215991ab"
server_key_version: "encryptionkey@0"
I20260619 06:44:31.727975 26176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:44:31.732201 26191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:31.733771 26176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:44:31.733950 26176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "4175df1c02154afaaa69f1d488cca890"
format_stamp: "Formatted at 2026-06-19 06:44:31 on dist-test-slave-qfk4"
server_key: "c14faf8de3fc28e458c1ecd392cf45bc"
server_key_iv: "07a5c93d8c584bb47d09122b215991ab"
server_key_version: "encryptionkey@0"
I20260619 06:44:31.734161 26176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:31.754264 26176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:31.758603 26176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:31.758881 26176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:31.759900 26176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:31.761488 26176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:31.761572 26176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:31.761693 26176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:31.761745 26176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:31.793113 26176 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:46301
I20260619 06:44:31.793160 26303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:46301 every 8 connection(s)
I20260619 06:44:31.794368 26176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:44:31.802106 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 26176
I20260619 06:44:31.802270 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:44:31.802706 7755 external_mini_cluster.cc:1428] Setting key eb6585a7c9d602ce72ebc6f9b8e56f96
Jun 19 06:44:31 dist-test-slave-qfk4 krb5kdc[25936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851471, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:31.814244 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:31.796976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:52611 (local address 127.7.146.254:37709)
0619 06:44:31.797164 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:31.797169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:31.798352 (+ 1183us) server_negotiation.cc:407] Connection header received
0619 06:44:31.800224 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:31.800229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:31.800327 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:31.800416 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:31.802012 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:31.802748 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:31.804500 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:31.804978 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:31.805104 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:31.808629 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:31.808634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:31.808638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:31.808672 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:31.811139 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:31.811852 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:31.811857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:31.811860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:31.811996 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:31.812410 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:31.812414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:31.812416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:31.812747 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:31.812883 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:31.813661 (+ 778us) server_negotiation.cc:299] Negotiation successful
0619 06:44:31.813909 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:44:31.815114 26306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:31.797502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:37709 (local address 127.7.146.194:52611)
0619 06:44:31.798186 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:31.798230 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:31.799950 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:31.800709 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:31.800718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:31.801257 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:31.801834 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:31.801852 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:31.802915 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:31.802920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:31.804303 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:31.804316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:31.805202 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:31.805210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:31.805434 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:31.806061 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:44:31.806089 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:31.808367 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:31.811271 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:31.811280 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:31.811299 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:31.811737 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:31.812105 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:31.812109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:31.812113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:31.812298 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:31.812865 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:31.812873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:31.813216 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 06:44:31.813531 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 06:44:31.813975 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":262,"threads_started":1}
I20260619 06:44:31.816546 26304 heartbeater.cc:347] Connected to a master server at 127.7.146.254:37709
I20260619 06:44:31.816874 26304 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:31.817641 26304 heartbeater.cc:511] Master 127.7.146.254:37709 requested a full tablet report, sending...
I20260619 06:44:31.819095 25981 ts_manager.cc:194] Registered new tserver with Master: 4175df1c02154afaaa69f1d488cca890 (127.7.146.194:46301)
I20260619 06:44:31.819947 25981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:52611
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:44:31.856487 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37709
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:32.172008 26309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:32.172423 26309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:32.172605 26309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:32.180656 26309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:32.180778 26309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:32.180899 26309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:44:32.190609 26309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:37709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:32.191805 26309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:32.193099 26309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:32.205771 26317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:32.205883 26315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:32.205909 26314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:32.206395 26309 server_base.cc:1034] running on GCE node
I20260619 06:44:32.207021 26309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:32.207958 26309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:32.209147 26309 hybrid_clock.cc:648] HybridClock initialized: now 1781851472209101 us; error 39 us; skew 500 ppm
I20260619 06:44:32.209438 26309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:32 dist-test-slave-qfk4 krb5kdc[25936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851472, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:32.212394 26309 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:44:32.213799 26309 webserver.cc:460] Webserver started at http://127.7.146.195:43383/ using document root <none> and password file <none>
I20260619 06:44:32.214450 26309 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:32.214557 26309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:32.214855 26309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:32.217803 26309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "20add9f3039144ca97436358b7af1f8b"
format_stamp: "Formatted at 2026-06-19 06:44:32 on dist-test-slave-qfk4"
server_key: "a6b5bf5430d2eeb8d7e74ca691ad026f"
server_key_iv: "8060e63ad3005d1d6f475ba38376a480"
server_key_version: "encryptionkey@0"
I20260619 06:44:32.218566 26309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "20add9f3039144ca97436358b7af1f8b"
format_stamp: "Formatted at 2026-06-19 06:44:32 on dist-test-slave-qfk4"
server_key: "a6b5bf5430d2eeb8d7e74ca691ad026f"
server_key_iv: "8060e63ad3005d1d6f475ba38376a480"
server_key_version: "encryptionkey@0"
I20260619 06:44:32.223685 26309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:44:32.227080 26324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:32.228394 26309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:44:32.228543 26309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "20add9f3039144ca97436358b7af1f8b"
format_stamp: "Formatted at 2026-06-19 06:44:32 on dist-test-slave-qfk4"
server_key: "a6b5bf5430d2eeb8d7e74ca691ad026f"
server_key_iv: "8060e63ad3005d1d6f475ba38376a480"
server_key_version: "encryptionkey@0"
I20260619 06:44:32.228760 26309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:32.249709 26309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:32.253829 26309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:32.254101 26309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:32.255031 26309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:32.256289 26309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:32.256361 26309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:32.256453 26309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:32.256508 26309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:32.288636 26309 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:38069
I20260619 06:44:32.288686 26436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:38069 every 8 connection(s)
I20260619 06:44:32.289898 26309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:44:32.298677 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 26309
I20260619 06:44:32.298859 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:44:32.299392 7755 external_mini_cluster.cc:1428] Setting key 8c9f957e1af8c492fdcd668cbb872845
Jun 19 06:44:32 dist-test-slave-qfk4 krb5kdc[25936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851472, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:32.311308 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.292325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:40049 (local address 127.7.146.254:37709)
0619 06:44:32.292496 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:32.292501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:32.293635 (+ 1134us) server_negotiation.cc:407] Connection header received
0619 06:44:32.295414 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:32.295420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:32.295508 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:32.295594 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:32.297368 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.298221 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.300320 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.300989 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.301111 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.304639 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:32.304644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:32.304648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:32.304681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:32.307944 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:32.308850 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:32.308856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:32.308860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:32.309033 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:32.309488 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:32.309493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:32.309496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:32.309862 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:32.310023 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:32.310709 (+ 686us) server_negotiation.cc:299] Negotiation successful
0619 06:44:32.310951 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:44:32.312009 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.292808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:37709 (local address 127.7.146.195:40049)
0619 06:44:32.293478 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:32.293518 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:32.295147 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:32.295831 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:32.295839 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:32.296550 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:32.297184 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.297206 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.298420 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.298425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.300156 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.300169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.301198 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.301207 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.301511 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.302257 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:44:32.302286 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:32.304411 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:32.308087 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:32.308096 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:32.308113 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:32.308716 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:32.309132 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:32.309137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:32.309141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:32.309369 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:32.309993 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:32.309999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:32.310306 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 06:44:32.310556 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 06:44:32.310936 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":250,"threads_started":1}
I20260619 06:44:32.313723 26437 heartbeater.cc:347] Connected to a master server at 127.7.146.254:37709
I20260619 06:44:32.314098 26437 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:32.314945 26437 heartbeater.cc:511] Master 127.7.146.254:37709 requested a full tablet report, sending...
I20260619 06:44:32.316553 25980 ts_manager.cc:194] Registered new tserver with Master: 20add9f3039144ca97436358b7af1f8b (127.7.146.195:38069)
I20260619 06:44:32.317495 25980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:40049
I20260619 06:44:32.326659 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:44:32.329768 26170 heartbeater.cc:503] Master 127.7.146.254:37709 was elected leader, sending a full tablet report...
I20260619 06:44:32.343518 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.330790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50268 (local address 127.7.146.254:37709)
0619 06:44:32.330940 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:32.330944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:32.331423 (+ 479us) server_negotiation.cc:407] Connection header received
0619 06:44:32.331585 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:32.331590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:32.331664 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:32.331741 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:32.333253 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.333997 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.335283 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.335699 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.335807 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.338412 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:32.338417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:32.338421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:32.338452 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:32.340964 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:32.341710 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:32.341715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:32.341719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:32.341844 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:32.342268 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:32.342272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:32.342274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:32.342590 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:32.342814 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:32.343159 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 06:44:32.343295 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:44:32.347426 25980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:44:32.354075 25980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:44:32.386266 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.365815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:44727 (local address 127.0.0.1:51558)
0619 06:44:32.377650 (+ 11835us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:32.377677 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:32.377802 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:32.379607 (+ 1805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:32.379615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:32.379642 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:32.379978 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.379987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.381383 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.381388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.383908 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.383917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.385227 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.385233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.385425 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.385474 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 06:44:32.385604 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 06:44:32.385815 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11683,"mutex_wait_us":23,"thread_start_us":12207,"threads_started":1}
I20260619 06:44:32.386917 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.366358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.7.146.193:44727)
0619 06:44:32.379067 (+ 12709us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:32.379075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:32.379101 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:44:32.379193 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:32.379199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:32.379404 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:32.379597 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:32.380117 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.381210 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.384049 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.385045 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.385167 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.385398 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:32.386528 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0619 06:44:32.386697 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12520,"thread_start_us":12906,"threads_started":1}
I20260619 06:44:32.387018 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.366665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:46301 (local address 127.0.0.1:44406)
0619 06:44:32.378250 (+ 11585us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:32.378271 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:32.378375 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:32.379292 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:32.379297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:32.379319 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:32.379651 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.379662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.381653 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.381658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.383656 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.383670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.386140 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.386146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.386401 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.386435 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:44:32.386514 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:44:32.386626 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11483,"mutex_wait_us":2,"thread_start_us":11303,"threads_started":1}
I20260619 06:44:32.387081 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.365174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:38069 (local address 127.0.0.1:44254)
0619 06:44:32.370836 (+ 5662us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:32.370894 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:32.371092 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:32.378276 (+ 7184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:32.378285 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:32.378321 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:32.378870 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.378884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.381139 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.381144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.383747 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.383761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.385780 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.385786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.385959 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.385989 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:44:32.386075 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 06:44:32.386217 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5384,"spinlock_wait_cycles":23424,"thread_start_us":605,"threads_started":1}
I20260619 06:44:32.387688 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.365335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.7.146.195:38069)
0619 06:44:32.377330 (+ 11995us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:32.377338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:32.377377 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:44:32.377499 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:32.377508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:32.377795 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:32.379069 (+ 1274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:32.379334 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.380985 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.383920 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.385267 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.385402 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.385628 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:32.386972 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0619 06:44:32.387395 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11792,"thread_start_us":11647,"threads_started":1}
I20260619 06:44:32.388113 26454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.367208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.7.146.194:46301)
0619 06:44:32.375536 (+ 8328us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:32.375546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:32.378502 (+ 2956us) server_negotiation.cc:407] Connection header received
0619 06:44:32.378607 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:32.378614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:32.378871 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:32.379070 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:32.379856 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.381344 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.383824 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.385673 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.385822 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.385994 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:32.387486 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0619 06:44:32.387646 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8183,"thread_start_us":8076,"threads_started":1}
I20260619 06:44:32.390913 26105 tablet_service.cc:1467] Processing CreateTablet for tablet cf3905e7beea4a009eea7ab8e0ba576a (DEFAULT_TABLE table=test-table [id=45b6a407cc11453099d0a66ef8930130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:32.391173 26372 tablet_service.cc:1467] Processing CreateTablet for tablet cf3905e7beea4a009eea7ab8e0ba576a (DEFAULT_TABLE table=test-table [id=45b6a407cc11453099d0a66ef8930130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:32.391396 26239 tablet_service.cc:1467] Processing CreateTablet for tablet cf3905e7beea4a009eea7ab8e0ba576a (DEFAULT_TABLE table=test-table [id=45b6a407cc11453099d0a66ef8930130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:32.392381 26372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3905e7beea4a009eea7ab8e0ba576a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:32.392432 26105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3905e7beea4a009eea7ab8e0ba576a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:32.392688 26239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3905e7beea4a009eea7ab8e0ba576a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:32.405221 26455 tablet_bootstrap.cc:492] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: Bootstrap starting.
I20260619 06:44:32.407035 26457 tablet_bootstrap.cc:492] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: Bootstrap starting.
I20260619 06:44:32.409204 26456 tablet_bootstrap.cc:492] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: Bootstrap starting.
I20260619 06:44:32.410775 26455 tablet_bootstrap.cc:654] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:32.410780 26457 tablet_bootstrap.cc:654] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:32.412035 26456 tablet_bootstrap.cc:654] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:32.412081 26455 log.cc:824] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:32.412081 26457 log.cc:824] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:32.413139 26456 log.cc:824] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:32.415136 26457 tablet_bootstrap.cc:492] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: No bootstrap required, opened a new log
I20260619 06:44:32.415190 26455 tablet_bootstrap.cc:492] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: No bootstrap required, opened a new log
I20260619 06:44:32.415393 26457 ts_tablet_manager.cc:1397] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 06:44:32.415517 26455 ts_tablet_manager.cc:1397] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260619 06:44:32.415922 26456 tablet_bootstrap.cc:492] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: No bootstrap required, opened a new log
I20260619 06:44:32.416178 26456 ts_tablet_manager.cc:1397] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:44:32.423156 26457 raft_consensus.cc:348] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20add9f3039144ca97436358b7af1f8b" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 38069 } } peers { permanent_uuid: "4175df1c02154afaaa69f1d488cca890" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46301 } } peers { permanent_uuid: "a061745ca9da4bad879ca1f3bf8791c6" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44727 } }
I20260619 06:44:32.423594 26457 raft_consensus.cc:374] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:32.423717 26457 raft_consensus.cc:729] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4175df1c02154afaaa69f1d488cca890, State: Initialized, Role: FOLLOWER
I20260619 06:44:32.424312 26457 consensus_queue.cc:260] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20add9f3039144ca97436358b7af1f8b" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 38069 } } peers { permanent_uuid: "4175df1c02154afaaa69f1d488cca890" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46301 } } peers { permanent_uuid: "a061745ca9da4bad879ca1f3bf8791c6" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44727 } }
I20260619 06:44:32.425566 26304 heartbeater.cc:503] Master 127.7.146.254:37709 was elected leader, sending a full tablet report...
I20260619 06:44:32.425926 26455 raft_consensus.cc:348] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20add9f3039144ca97436358b7af1f8b" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 38069 } } peers { permanent_uuid: "4175df1c02154afaaa69f1d488cca890" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46301 } } peers { permanent_uuid: "a061745ca9da4bad879ca1f3bf8791c6" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44727 } }
I20260619 06:44:32.426362 26455 raft_consensus.cc:374] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:32.426502 26455 raft_consensus.cc:729] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20add9f3039144ca97436358b7af1f8b, State: Initialized, Role: FOLLOWER
I20260619 06:44:32.426515 26456 raft_consensus.cc:348] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20add9f3039144ca97436358b7af1f8b" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 38069 } } peers { permanent_uuid: "4175df1c02154afaaa69f1d488cca890" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46301 } } peers { permanent_uuid: "a061745ca9da4bad879ca1f3bf8791c6" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44727 } }
I20260619 06:44:32.426982 26456 raft_consensus.cc:374] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:32.427099 26456 raft_consensus.cc:729] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a061745ca9da4bad879ca1f3bf8791c6, State: Initialized, Role: FOLLOWER
I20260619 06:44:32.427114 26455 consensus_queue.cc:260] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20add9f3039144ca97436358b7af1f8b" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 38069 } } peers { permanent_uuid: "4175df1c02154afaaa69f1d488cca890" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46301 } } peers { permanent_uuid: "a061745ca9da4bad879ca1f3bf8791c6" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44727 } }
I20260619 06:44:32.427672 26456 consensus_queue.cc:260] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20add9f3039144ca97436358b7af1f8b" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 38069 } } peers { permanent_uuid: "4175df1c02154afaaa69f1d488cca890" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46301 } } peers { permanent_uuid: "a061745ca9da4bad879ca1f3bf8791c6" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44727 } }
I20260619 06:44:32.428078 26457 ts_tablet_manager.cc:1428] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260619 06:44:32.429558 26437 heartbeater.cc:503] Master 127.7.146.254:37709 was elected leader, sending a full tablet report...
I20260619 06:44:32.429828 26455 ts_tablet_manager.cc:1428] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 06:44:32.430073 26456 ts_tablet_manager.cc:1428] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260619 06:44:32.515625 26463 raft_consensus.cc:482] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:32.515965 26463 raft_consensus.cc:504] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20add9f3039144ca97436358b7af1f8b" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 38069 } } peers { permanent_uuid: "4175df1c02154afaaa69f1d488cca890" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46301 } } peers { permanent_uuid: "a061745ca9da4bad879ca1f3bf8791c6" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44727 } }
I20260619 06:44:32.517627 26463 leader_election.cc:288] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20add9f3039144ca97436358b7af1f8b (127.7.146.195:38069), 4175df1c02154afaaa69f1d488cca890 (127.7.146.194:46301)
I20260619 06:44:32.524668 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.518083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:46455 (local address 127.7.146.195:38069)
0619 06:44:32.518257 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:32.518261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:32.518865 (+ 604us) server_negotiation.cc:407] Connection header received
0619 06:44:32.519108 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:32.519113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:32.519192 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:32.519298 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:32.519881 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.520871 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.522746 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.523695 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.523820 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.523873 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:32.524264 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 06:44:32.524396 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:44:32.524699 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.518054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:38069 (local address 127.7.146.193:46455)
0619 06:44:32.518739 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:32.518757 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:32.518898 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:32.519358 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:32.519363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:32.519398 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:32.519724 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.519737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.520990 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.520995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.522600 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.522615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.523895 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.523902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.524066 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.524128 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 06:44:32.524222 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:44:32.524324 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":355,"threads_started":1}
I20260619 06:44:32.524897 26454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.519062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:39091 (local address 127.7.146.194:46301)
0619 06:44:32.519246 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:32.519251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:32.519580 (+ 329us) server_negotiation.cc:407] Connection header received
0619 06:44:32.519697 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:32.519701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:32.519773 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:32.519863 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:32.520421 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.521354 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.523106 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.523969 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.524111 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.524179 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:32.524513 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 06:44:32.524676 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:44:32.525000 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.518884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:46301 (local address 127.7.146.193:39091)
0619 06:44:32.519477 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:32.519495 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:32.519597 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:32.519888 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:32.519892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:32.519916 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:32.520275 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.520287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.521513 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.521517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.522967 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:32.522981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.524175 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.524180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:32.524348 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.524381 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:44:32.524485 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:44:32.524635 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":1,"thread_start_us":303,"threads_started":1}
I20260619 06:44:32.525904 26259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3905e7beea4a009eea7ab8e0ba576a" candidate_uuid: "a061745ca9da4bad879ca1f3bf8791c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4175df1c02154afaaa69f1d488cca890" is_pre_election: true
I20260619 06:44:32.525970 26392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3905e7beea4a009eea7ab8e0ba576a" candidate_uuid: "a061745ca9da4bad879ca1f3bf8791c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20add9f3039144ca97436358b7af1f8b" is_pre_election: true
I20260619 06:44:32.526382 26259 raft_consensus.cc:2454] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a061745ca9da4bad879ca1f3bf8791c6 in term 0.
I20260619 06:44:32.526422 26392 raft_consensus.cc:2454] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a061745ca9da4bad879ca1f3bf8791c6 in term 0.
I20260619 06:44:32.527004 26059 leader_election.cc:302] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4175df1c02154afaaa69f1d488cca890, a061745ca9da4bad879ca1f3bf8791c6; no voters:
I20260619 06:44:32.527360 26463 raft_consensus.cc:2781] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:44:32.527525 26463 raft_consensus.cc:482] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:44:32.527627 26463 raft_consensus.cc:3037] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:32.529866 26463 raft_consensus.cc:504] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20add9f3039144ca97436358b7af1f8b" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 38069 } } peers { permanent_uuid: "4175df1c02154afaaa69f1d488cca890" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46301 } } peers { permanent_uuid: "a061745ca9da4bad879ca1f3bf8791c6" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44727 } }
I20260619 06:44:32.530408 26463 leader_election.cc:288] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [CANDIDATE]: Term 1 election: Requested vote from peers 20add9f3039144ca97436358b7af1f8b (127.7.146.195:38069), 4175df1c02154afaaa69f1d488cca890 (127.7.146.194:46301)
I20260619 06:44:32.530879 26392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3905e7beea4a009eea7ab8e0ba576a" candidate_uuid: "a061745ca9da4bad879ca1f3bf8791c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20add9f3039144ca97436358b7af1f8b"
I20260619 06:44:32.530886 26259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3905e7beea4a009eea7ab8e0ba576a" candidate_uuid: "a061745ca9da4bad879ca1f3bf8791c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4175df1c02154afaaa69f1d488cca890"
I20260619 06:44:32.531054 26392 raft_consensus.cc:3037] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:32.531066 26259 raft_consensus.cc:3037] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:32.533396 26392 raft_consensus.cc:2454] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a061745ca9da4bad879ca1f3bf8791c6 in term 1.
I20260619 06:44:32.533494 26259 raft_consensus.cc:2454] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a061745ca9da4bad879ca1f3bf8791c6 in term 1.
I20260619 06:44:32.533840 26059 leader_election.cc:302] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20add9f3039144ca97436358b7af1f8b, a061745ca9da4bad879ca1f3bf8791c6; no voters:
I20260619 06:44:32.534111 26463 raft_consensus.cc:2781] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:32.534586 26463 raft_consensus.cc:686] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 1 LEADER]: Becoming Leader. State: Replica: a061745ca9da4bad879ca1f3bf8791c6, State: Running, Role: LEADER
I20260619 06:44:32.535122 26463 consensus_queue.cc:237] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20add9f3039144ca97436358b7af1f8b" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 38069 } } peers { permanent_uuid: "4175df1c02154afaaa69f1d488cca890" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46301 } } peers { permanent_uuid: "a061745ca9da4bad879ca1f3bf8791c6" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44727 } }
I20260619 06:44:32.539809 25981 catalog_manager.cc:5495] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a061745ca9da4bad879ca1f3bf8791c6 (127.7.146.193). New cstate: current_term: 1 leader_uuid: "a061745ca9da4bad879ca1f3bf8791c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20add9f3039144ca97436358b7af1f8b" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 38069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4175df1c02154afaaa69f1d488cca890" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a061745ca9da4bad879ca1f3bf8791c6" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 44727 } health_report { overall_health: HEALTHY } } }
W20260619 06:44:32.541359 26438 tablet.cc:2369] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:44:32.546275 26305 tablet.cc:2369] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:44:32.551904 26171 tablet.cc:2369] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:44:32.584543 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.579079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.7.146.193:44727)
0619 06:44:32.579253 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:32.579257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:32.579274 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:44:32.579425 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:32.579429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:32.579512 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:32.579633 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:44:32.579998 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.580747 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.582187 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.582588 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.582679 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.583309 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:44:32.583934 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:44:32.584109 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:32.584291 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 06:44:32.584334 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:44:32.593178 26392 raft_consensus.cc:1261] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b [term 1 FOLLOWER]: Refusing update from remote peer a061745ca9da4bad879ca1f3bf8791c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:32.593178 26259 raft_consensus.cc:1261] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890 [term 1 FOLLOWER]: Refusing update from remote peer a061745ca9da4bad879ca1f3bf8791c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:32.594211 26466 consensus_queue.cc:1035] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4175df1c02154afaaa69f1d488cca890" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:32.594580 26463 consensus_queue.cc:1035] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20add9f3039144ca97436358b7af1f8b" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:32.607069 26469 mvcc.cc:205] Tried to move back new op lower bound from 7298463631730335744 to 7298463631507722240. Current Snapshot: MvccSnapshot[applied={T|T < 7298463631730335744}]
I20260619 06:44:32.608728 26471 mvcc.cc:205] Tried to move back new op lower bound from 7298463631730335744 to 7298463631507722240. Current Snapshot: MvccSnapshot[applied={T|T < 7298463631730335744}]
I20260619 06:44:32.609126 26470 mvcc.cc:205] Tried to move back new op lower bound from 7298463631730335744 to 7298463631507722240. Current Snapshot: MvccSnapshot[applied={T|T < 7298463631730335744}]
I20260619 06:44:32.624846 25981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:32.625269 25981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:32.629194 25981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 918cf0e6087043178178fb194c7b223a: Sending DeleteTablet for 3 replicas of tablet cf3905e7beea4a009eea7ab8e0ba576a
I20260619 06:44:32.630338 26372 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3905e7beea4a009eea7ab8e0ba576a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:32 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:44254
I20260619 06:44:32.630375 26239 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3905e7beea4a009eea7ab8e0ba576a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:32 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:44406
I20260619 06:44:32.631071 26105 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3905e7beea4a009eea7ab8e0ba576a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:32 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:51558
I20260619 06:44:32.631951 26481 tablet_replica.cc:330] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: stopping tablet replica
I20260619 06:44:32.632412 26481 raft_consensus.cc:2229] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:44:32.633082 26481 raft_consensus.cc:2258] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:32.635210 26481 ts_tablet_manager.cc:1905] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:32.635425 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 26042
I20260619 06:44:32.642489 26479 tablet_replica.cc:330] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: stopping tablet replica
I20260619 06:44:32.642911 26481 ts_tablet_manager.cc:1918] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:32.643083 26481 log.cc:1196] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/cf3905e7beea4a009eea7ab8e0ba576a
I20260619 06:44:32.643354 26479 raft_consensus.cc:2229] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:32.643625 26481 ts_tablet_manager.cc:1939] T cf3905e7beea4a009eea7ab8e0ba576a P a061745ca9da4bad879ca1f3bf8791c6: Deleting consensus metadata
I20260619 06:44:32.644021 26479 raft_consensus.cc:2258] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:32.645429 25967 catalog_manager.cc:4841] TS a061745ca9da4bad879ca1f3bf8791c6 (127.7.146.193:44727): tablet cf3905e7beea4a009eea7ab8e0ba576a (table test-table [id=45b6a407cc11453099d0a66ef8930130]) successfully deleted
I20260619 06:44:32.646375 26479 ts_tablet_manager.cc:1905] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:32.653149 26480 tablet_replica.cc:330] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: stopping tablet replica
I20260619 06:44:32.653609 26479 ts_tablet_manager.cc:1918] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:32.653613 26480 raft_consensus.cc:2229] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:32.653852 26479 log.cc:1196] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/cf3905e7beea4a009eea7ab8e0ba576a
I20260619 06:44:32.654150 26480 raft_consensus.cc:2258] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:32.654477 26479 ts_tablet_manager.cc:1939] T cf3905e7beea4a009eea7ab8e0ba576a P 4175df1c02154afaaa69f1d488cca890: Deleting consensus metadata
I20260619 06:44:32.656325 25966 catalog_manager.cc:4841] TS 4175df1c02154afaaa69f1d488cca890 (127.7.146.194:46301): tablet cf3905e7beea4a009eea7ab8e0ba576a (table test-table [id=45b6a407cc11453099d0a66ef8930130]) successfully deleted
I20260619 06:44:32.656448 26480 ts_tablet_manager.cc:1905] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 06:44:32 dist-test-slave-qfk4 krb5kdc[25936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:44:32.663280 26480 ts_tablet_manager.cc:1918] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:32.663453 26480 log.cc:1196] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/cf3905e7beea4a009eea7ab8e0ba576a
I20260619 06:44:32.663998 26480 ts_tablet_manager.cc:1939] T cf3905e7beea4a009eea7ab8e0ba576a P 20add9f3039144ca97436358b7af1f8b: Deleting consensus metadata
I20260619 06:44:32.665884 25966 catalog_manager.cc:4841] TS 20add9f3039144ca97436358b7af1f8b (127.7.146.195:38069): tablet cf3905e7beea4a009eea7ab8e0ba576a (table test-table [id=45b6a407cc11453099d0a66ef8930130]) successfully deleted
I20260619 06:44:32.669595 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.636504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.7.146.193:44727)
0619 06:44:32.636697 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:32.636703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:32.654724 (+ 18021us) server_negotiation.cc:407] Connection header received
0619 06:44:32.655239 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:32.655245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:32.655329 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:32.655410 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:32.657128 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.657914 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.659197 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.659669 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.659774 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.663822 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:32.663837 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:32.663851 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:32.663885 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:32.666907 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:32.667578 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:32.667585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:32.667588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:32.667724 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:32.668098 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:32.668102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:32.668104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:32.668464 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:32.668802 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:32.669037 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:44:32.669265 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:44:32.670554 26165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:32.771325 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 26042
I20260619 06:44:32.786053 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 26176
Jun 19 06:44:32 dist-test-slave-qfk4 krb5kdc[25936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:44:32.800683 26454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.786825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.7.146.194:46301)
0619 06:44:32.787053 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:32.787059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:32.787078 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:44:32.787140 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:32.787144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:32.787206 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:32.787289 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:32.788508 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.789279 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.790515 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.790919 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.791039 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.795174 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:32.795189 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:32.795203 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:32.795239 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:32.798023 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:32.798667 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:32.798674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:32.798676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:32.798801 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:32.799139 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:32.799143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:32.799145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:32.799528 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:32.799791 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:32.800088 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 06:44:32.800314 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 06:44:32.801437 26299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:32.890049 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 26176
I20260619 06:44:32.904343 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 26309
Jun 19 06:44:32 dist-test-slave-qfk4 krb5kdc[25936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:44:32.919091 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:32.905153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.7.146.195:38069)
0619 06:44:32.905354 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:32.905360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:32.905380 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:44:32.905500 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:32.905504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:32.905569 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:32.905652 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:32.906996 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.907731 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.909113 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:32.909516 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:32.909626 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:32.913780 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:32.913795 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:32.913811 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:32.913855 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:32.916299 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:32.917040 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:32.917050 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:32.917054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:32.917195 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:32.917552 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:32.917557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:32.917559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:32.917915 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:32.918191 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:32.918513 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 06:44:32.918774 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:44:32.919701 26432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:33.007982 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 26309
I20260619 06:44:33.020937 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 25949
I20260619 06:44:33.022059 26011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:33.133407 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 25949
2026-06-19T06:44:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4137 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T06:44:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:33Z Disabled control of system clock
I20260619 06:44:33.202486 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:35453
--webserver_interface=127.7.146.254
--webserver_port=0
--builtin_ntp_servers=127.7.146.212:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:35453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:44:33.524227 26490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:33.524524 26490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:33.524652 26490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:33.531810 26490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:33.531924 26490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:33.531978 26490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:33.532056 26490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:33.532104 26490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:33.541352 26490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:35453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:35453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:33.542682 26490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:33.544430 26490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:33.555773 26498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:33.555809 26495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:33.555905 26496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:33.556510 26490 server_base.cc:1034] running on GCE node
I20260619 06:44:33.557272 26490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:33.558655 26490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:33.559870 26490 hybrid_clock.cc:648] HybridClock initialized: now 1781851473559810 us; error 40 us; skew 500 ppm
I20260619 06:44:33.560218 26490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:33.562650 26490 webserver.cc:460] Webserver started at http://127.7.146.254:41755/ using document root <none> and password file <none>
I20260619 06:44:33.563449 26490 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:33.563551 26490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:33.563900 26490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:33.567015 26490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "f7ffaabe6c9444e69b4ff841be31a278"
format_stamp: "Formatted at 2026-06-19 06:44:33 on dist-test-slave-qfk4"
server_key: "429b506fdcbb4056cda930f58c3237d0"
server_key_iv: "a054da67beee55cdda54e422cf1c32ec"
server_key_version: "encryptionkey@0"
I20260619 06:44:33.567888 26490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "f7ffaabe6c9444e69b4ff841be31a278"
format_stamp: "Formatted at 2026-06-19 06:44:33 on dist-test-slave-qfk4"
server_key: "429b506fdcbb4056cda930f58c3237d0"
server_key_iv: "a054da67beee55cdda54e422cf1c32ec"
server_key_version: "encryptionkey@0"
I20260619 06:44:33.573541 26490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:44:33.577342 26504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:33.578917 26490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:44:33.579093 26490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "f7ffaabe6c9444e69b4ff841be31a278"
format_stamp: "Formatted at 2026-06-19 06:44:33 on dist-test-slave-qfk4"
server_key: "429b506fdcbb4056cda930f58c3237d0"
server_key_iv: "a054da67beee55cdda54e422cf1c32ec"
server_key_version: "encryptionkey@0"
I20260619 06:44:33.579298 26490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:33.592312 26490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:33.593173 26490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:33.593451 26490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:33.610206 26490 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:35453
I20260619 06:44:33.610224 26555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:35453 every 8 connection(s)
I20260619 06:44:33.611608 26490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:33.613948 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 26490
I20260619 06:44:33.614104 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:33.614542 7755 external_mini_cluster.cc:1428] Setting key 68b17a45f6916a7ce7831adfa6181dfa
I20260619 06:44:33.615372 26556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:33.629767 26556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278: Bootstrap starting.
I20260619 06:44:33.630096 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:33.616430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.7.146.254:35453)
0619 06:44:33.617508 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:33.617523 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:33.617572 (+ 49us) server_negotiation.cc:407] Connection header received
0619 06:44:33.619943 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:33.619984 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:33.620630 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:33.621092 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:33.621633 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:33.623545 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:33.625086 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:33.626104 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:33.626281 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:33.627353 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:33.627364 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:33.627382 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:33.628070 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:33.628163 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:33.628188 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:33.628301 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:33.628500 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 06:44:33.628835 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":363,"threads_started":1}
I20260619 06:44:33.633335 26556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:33.634241 26556 log.cc:824] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:33.636721 26556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278: No bootstrap required, opened a new log
I20260619 06:44:33.643927 26556 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ffaabe6c9444e69b4ff841be31a278" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35453 } }
I20260619 06:44:33.644256 26556 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:33.644326 26556 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ffaabe6c9444e69b4ff841be31a278, State: Initialized, Role: FOLLOWER
I20260619 06:44:33.644857 26556 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ffaabe6c9444e69b4ff841be31a278" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35453 } }
I20260619 06:44:33.645056 26556 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:33.645146 26556 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:33.645264 26556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:33.647663 26556 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ffaabe6c9444e69b4ff841be31a278" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35453 } }
I20260619 06:44:33.648134 26556 leader_election.cc:302] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7ffaabe6c9444e69b4ff841be31a278; no voters:
I20260619 06:44:33.648685 26556 leader_election.cc:288] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:33.648816 26561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:33.649508 26561 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [term 1 LEADER]: Becoming Leader. State: Replica: f7ffaabe6c9444e69b4ff841be31a278, State: Running, Role: LEADER
I20260619 06:44:33.650012 26561 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ffaabe6c9444e69b4ff841be31a278" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35453 } }
I20260619 06:44:33.650563 26556 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:33.652794 26563 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7ffaabe6c9444e69b4ff841be31a278. Latest consensus state: current_term: 1 leader_uuid: "f7ffaabe6c9444e69b4ff841be31a278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ffaabe6c9444e69b4ff841be31a278" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35453 } } }
I20260619 06:44:33.653236 26563 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:33.653394 26562 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7ffaabe6c9444e69b4ff841be31a278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ffaabe6c9444e69b4ff841be31a278" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35453 } } }
I20260619 06:44:33.653712 26562 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:33.661834 26570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:33.666455 26570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:33.674733 26570 catalog_manager.cc:1347] Generated new cluster ID: 1465a52f81b448f194257b1804f6468c
I20260619 06:44:33.674901 26570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:33.687427 26570 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:33.688705 26570 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:33.703987 26570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278: Generated new TSK 0
I20260619 06:44:33.704916 26570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:33.713541 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35453
--builtin_ntp_servers=127.7.146.212:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:44:34.036389 26580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:34.036738 26580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:34.036851 26580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:34.044144 26580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:34.044241 26580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:34.044351 26580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:44:34.053867 26580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:34.055171 26580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:34.056912 26580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:34.069389 26588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:34.069453 26586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:34.069432 26585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:34.070281 26580 server_base.cc:1034] running on GCE node
I20260619 06:44:34.070963 26580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:34.072055 26580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:34.073298 26580 hybrid_clock.cc:648] HybridClock initialized: now 1781851474073234 us; error 53 us; skew 500 ppm
I20260619 06:44:34.073621 26580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:34.075909 26580 webserver.cc:460] Webserver started at http://127.7.146.193:35673/ using document root <none> and password file <none>
I20260619 06:44:34.076614 26580 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:34.076725 26580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:34.077013 26580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:34.079869 26580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "972023942a57402e8ae2ec81d4659fc0"
format_stamp: "Formatted at 2026-06-19 06:44:34 on dist-test-slave-qfk4"
server_key: "64616b9482085aab8e9ddee0e88bc023"
server_key_iv: "8d7970e74e5e6b1f6d844d78bc003623"
server_key_version: "encryptionkey@0"
I20260619 06:44:34.080670 26580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "972023942a57402e8ae2ec81d4659fc0"
format_stamp: "Formatted at 2026-06-19 06:44:34 on dist-test-slave-qfk4"
server_key: "64616b9482085aab8e9ddee0e88bc023"
server_key_iv: "8d7970e74e5e6b1f6d844d78bc003623"
server_key_version: "encryptionkey@0"
I20260619 06:44:34.086022 26580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 06:44:34.090240 26594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:34.091774 26580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260619 06:44:34.092008 26580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "972023942a57402e8ae2ec81d4659fc0"
format_stamp: "Formatted at 2026-06-19 06:44:34 on dist-test-slave-qfk4"
server_key: "64616b9482085aab8e9ddee0e88bc023"
server_key_iv: "8d7970e74e5e6b1f6d844d78bc003623"
server_key_version: "encryptionkey@0"
I20260619 06:44:34.092240 26580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:34.109025 26580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:34.109869 26580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:34.110142 26580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:34.111289 26580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:34.112947 26580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:34.113034 26580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:34.113135 26580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:34.113204 26580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:34.142627 26580 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:37023
I20260619 06:44:34.142673 26706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:37023 every 8 connection(s)
I20260619 06:44:34.143903 26580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:44:34.145660 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 26580
I20260619 06:44:34.145829 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:44:34.146286 7755 external_mini_cluster.cc:1428] Setting key 4e4b41bea8227081a4b7f4cac2a1ea09
I20260619 06:44:34.149466 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35453
--builtin_ntp_servers=127.7.146.212:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:44:34.158162 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:34.146589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:51407 (local address 127.7.146.254:35453)
0619 06:44:34.147067 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:34.147073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:34.148311 (+ 1238us) server_negotiation.cc:407] Connection header received
0619 06:44:34.150796 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:34.150801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:34.150878 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:34.150995 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:34.152369 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:34.153180 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:34.155303 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:34.155719 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:34.155818 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:34.157297 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:34.157301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:34.157305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:34.157322 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:34.157376 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:34.157389 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:34.157455 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:34.157714 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 06:44:34.157819 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260619 06:44:34.159003 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:34.147177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:35453 (local address 127.7.146.193:51407)
0619 06:44:34.148173 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:34.148212 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:34.150475 (+ 2263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:34.151257 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:34.151268 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:34.151386 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:34.152170 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:34.152195 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:34.153338 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:34.153343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:34.155149 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:34.155163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:34.155928 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:34.155934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:34.156204 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:34.156944 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:44:34.156947 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:34.156998 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:44:34.157097 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:34.157480 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:34.157492 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:34.157500 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:44:34.157661 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 06:44:34.157957 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":305,"threads_started":1}
I20260619 06:44:34.160552 26707 heartbeater.cc:347] Connected to a master server at 127.7.146.254:35453
I20260619 06:44:34.160848 26707 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:34.161623 26707 heartbeater.cc:511] Master 127.7.146.254:35453 requested a full tablet report, sending...
I20260619 06:44:34.163833 26521 ts_manager.cc:194] Registered new tserver with Master: 972023942a57402e8ae2ec81d4659fc0 (127.7.146.193:37023)
I20260619 06:44:34.165836 26521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.193:51407
W20260619 06:44:34.472084 26711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:34.472368 26711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:34.472474 26711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:34.480010 26711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:34.480145 26711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:34.480297 26711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:44:34.490341 26711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:34.491662 26711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:34.493448 26711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:34.506192 26716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:34.506219 26719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:34.506166 26717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:34.506871 26711 server_base.cc:1034] running on GCE node
I20260619 06:44:34.507411 26711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:34.508287 26711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:34.509487 26711 hybrid_clock.cc:648] HybridClock initialized: now 1781851474509438 us; error 36 us; skew 500 ppm
I20260619 06:44:34.509784 26711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:34.511878 26711 webserver.cc:460] Webserver started at http://127.7.146.194:38753/ using document root <none> and password file <none>
I20260619 06:44:34.512524 26711 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:34.512650 26711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:34.512954 26711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:34.515723 26711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "c310b86d31374f99a39d0545f945441c"
format_stamp: "Formatted at 2026-06-19 06:44:34 on dist-test-slave-qfk4"
server_key: "61932179b741dc84ba20a846712a90c3"
server_key_iv: "3ed0467b01d3605ccfabe5d7f223b09e"
server_key_version: "encryptionkey@0"
I20260619 06:44:34.516453 26711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "c310b86d31374f99a39d0545f945441c"
format_stamp: "Formatted at 2026-06-19 06:44:34 on dist-test-slave-qfk4"
server_key: "61932179b741dc84ba20a846712a90c3"
server_key_iv: "3ed0467b01d3605ccfabe5d7f223b09e"
server_key_version: "encryptionkey@0"
I20260619 06:44:34.521421 26711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 06:44:34.524854 26725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:34.526162 26711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:44:34.526314 26711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "c310b86d31374f99a39d0545f945441c"
format_stamp: "Formatted at 2026-06-19 06:44:34 on dist-test-slave-qfk4"
server_key: "61932179b741dc84ba20a846712a90c3"
server_key_iv: "3ed0467b01d3605ccfabe5d7f223b09e"
server_key_version: "encryptionkey@0"
I20260619 06:44:34.526538 26711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:34.539670 26711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:34.540477 26711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:34.540787 26711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:34.541680 26711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:34.543072 26711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:34.543165 26711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:34.543273 26711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:34.543341 26711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:34.572973 26711 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:38557
I20260619 06:44:34.573028 26837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:38557 every 8 connection(s)
I20260619 06:44:34.574254 26711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:44:34.575711 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 26711
I20260619 06:44:34.575873 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:44:34.576304 7755 external_mini_cluster.cc:1428] Setting key 4bb90b539d6bf6ae900a826c5b00bae9
I20260619 06:44:34.579202 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35453
--builtin_ntp_servers=127.7.146.212:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:44:34.586853 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:34.576696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:53417 (local address 127.7.146.254:35453)
0619 06:44:34.576987 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:34.576993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:34.578559 (+ 1566us) server_negotiation.cc:407] Connection header received
0619 06:44:34.580433 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:34.580438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:34.580536 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:34.580653 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:34.581749 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:34.582492 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:34.584156 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:34.584633 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:34.584758 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:34.586158 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:34.586163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:34.586166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:34.586185 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:34.586219 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:34.586232 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:34.586306 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:34.586533 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 06:44:34.586620 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260619 06:44:34.588241 26840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:34.577341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:35453 (local address 127.7.146.194:53417)
0619 06:44:34.578407 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:34.578455 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:34.580114 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:34.580902 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:34.580911 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:34.581006 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:34.581578 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:34.581595 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:34.582657 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:34.582661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:34.584022 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:34.584034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:34.584862 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:34.584870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:34.585090 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:34.585775 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:44:34.585779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:34.585834 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:44:34.585954 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:34.586328 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:34.586338 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:34.586346 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:44:34.586525 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 06:44:34.586964 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":295,"threads_started":1}
I20260619 06:44:34.589694 26838 heartbeater.cc:347] Connected to a master server at 127.7.146.254:35453
I20260619 06:44:34.590022 26838 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:34.590912 26838 heartbeater.cc:511] Master 127.7.146.254:35453 requested a full tablet report, sending...
I20260619 06:44:34.592392 26521 ts_manager.cc:194] Registered new tserver with Master: c310b86d31374f99a39d0545f945441c (127.7.146.194:38557)
I20260619 06:44:34.593233 26521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.194:53417
W20260619 06:44:34.902066 26841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:34.902354 26841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:34.902463 26841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:34.909776 26841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:34.909909 26841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:34.910061 26841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:44:34.919209 26841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:35453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:34.920454 26841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:34.922251 26841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:34.936717 26849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:34.936700 26847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:34.936828 26846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:34.938871 26841 server_base.cc:1034] running on GCE node
I20260619 06:44:34.939602 26841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:34.940557 26841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:34.941800 26841 hybrid_clock.cc:648] HybridClock initialized: now 1781851474941747 us; error 49 us; skew 500 ppm
I20260619 06:44:34.942107 26841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:34.944310 26841 webserver.cc:460] Webserver started at http://127.7.146.195:46125/ using document root <none> and password file <none>
I20260619 06:44:34.945019 26841 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:34.945114 26841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:34.945408 26841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:34.948251 26841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "0e134add5f964603bf46200d4b586ee2"
format_stamp: "Formatted at 2026-06-19 06:44:34 on dist-test-slave-qfk4"
server_key: "f66b5f353fa46a40477d6e3e84bb9bf8"
server_key_iv: "0d8c85f12e8cc291f0d1b1809d2f5826"
server_key_version: "encryptionkey@0"
I20260619 06:44:34.949115 26841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "0e134add5f964603bf46200d4b586ee2"
format_stamp: "Formatted at 2026-06-19 06:44:34 on dist-test-slave-qfk4"
server_key: "f66b5f353fa46a40477d6e3e84bb9bf8"
server_key_iv: "0d8c85f12e8cc291f0d1b1809d2f5826"
server_key_version: "encryptionkey@0"
I20260619 06:44:34.954591 26841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:44:34.958281 26855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:34.959848 26841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:44:34.960063 26841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "0e134add5f964603bf46200d4b586ee2"
format_stamp: "Formatted at 2026-06-19 06:44:34 on dist-test-slave-qfk4"
server_key: "f66b5f353fa46a40477d6e3e84bb9bf8"
server_key_iv: "0d8c85f12e8cc291f0d1b1809d2f5826"
server_key_version: "encryptionkey@0"
I20260619 06:44:34.960278 26841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:34.991498 26841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:34.992348 26841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:34.992630 26841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:34.993587 26841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:34.994987 26841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:34.995069 26841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:34.995147 26841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:34.995198 26841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:35.025098 26841 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:44137
I20260619 06:44:35.025169 26967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:44137 every 8 connection(s)
I20260619 06:44:35.026371 26841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:44:35.035187 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 26841
I20260619 06:44:35.035354 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:44:35.035794 7755 external_mini_cluster.cc:1428] Setting key dc41751f158e406a6d574414ae91b1d2
I20260619 06:44:35.041057 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.028978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:51585 (local address 127.7.146.254:35453)
0619 06:44:35.029201 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:35.029208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:35.030953 (+ 1745us) server_negotiation.cc:407] Connection header received
0619 06:44:35.033604 (+ 2651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:35.033611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:35.033709 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:35.033831 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:35.035117 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.035960 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.038255 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.038772 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.038899 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.040238 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:35.040244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:35.040247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:35.040265 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:35.040298 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:35.040309 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:35.040381 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:35.040691 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 06:44:35.040808 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:44:35.042110 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.029562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:35453 (local address 127.7.146.195:51585)
0619 06:44:35.030700 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:35.030825 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:35.033225 (+ 2400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:35.034153 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:35.034163 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:35.034272 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:35.034903 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.034923 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.036118 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.036125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.038095 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.038119 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.038993 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.038999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.039221 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.039922 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:44:35.039924 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:35.039968 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:44:35.040056 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:35.040424 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:35.040434 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:35.040441 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:44:35.040685 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 06:44:35.041064 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":376,"threads_started":1}
I20260619 06:44:35.043363 26968 heartbeater.cc:347] Connected to a master server at 127.7.146.254:35453
I20260619 06:44:35.043645 26968 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:35.044378 26968 heartbeater.cc:511] Master 127.7.146.254:35453 requested a full tablet report, sending...
I20260619 06:44:35.046017 26521 ts_manager.cc:194] Registered new tserver with Master: 0e134add5f964603bf46200d4b586ee2 (127.7.146.195:44137)
I20260619 06:44:35.046949 26521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.195:51585
I20260619 06:44:35.051247 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:44:35.060950 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.055404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.7.146.254:35453)
0619 06:44:35.055572 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:35.055577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:35.055973 (+ 396us) server_negotiation.cc:407] Connection header received
0619 06:44:35.056113 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:35.056116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:35.056177 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:35.056258 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:35.056736 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.057429 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.058823 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.059235 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.059341 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.060312 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:35.060317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:35.060320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:35.060337 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:35.060370 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:35.060382 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:35.060509 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:35.060664 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 06:44:35.060713 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:44:35.064661 26521 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:44:35.071249 26521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:44:35.103003 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.083832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:44137 (local address 127.0.0.1:46942)
0619 06:44:35.095658 (+ 11826us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:35.095676 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:35.095918 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:35.096624 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:35.096628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:35.096656 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:35.097042 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.097051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.098503 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.098508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.100409 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.100427 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.102112 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.102117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.102266 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.102302 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:44:35.102423 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 06:44:35.102608 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11700,"mutex_wait_us":4,"thread_start_us":11425,"threads_started":1}
I20260619 06:44:35.103029 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.084531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.7.146.195:44137)
0619 06:44:35.094679 (+ 10148us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:35.094687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:35.095771 (+ 1084us) server_negotiation.cc:407] Connection header received
0619 06:44:35.096098 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:35.096105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:35.096328 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:35.096500 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:35.097205 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.098346 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.100666 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.101917 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.102053 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.102333 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:35.102535 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 06:44:35.102700 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9963,"thread_start_us":10275,"threads_started":1}
I20260619 06:44:35.104137 26984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.083140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:38557 (local address 127.0.0.1:38660)
0619 06:44:35.095578 (+ 12438us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:35.095600 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:35.095715 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:35.096357 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:35.096361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:35.096385 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:35.096863 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.096877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.098676 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.098681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.101297 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.101307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.103341 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.103347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.103547 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.103580 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:44:35.103658 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 06:44:35.103786 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12288,"mutex_wait_us":114,"thread_start_us":12302,"threads_started":1}
I20260619 06:44:35.104941 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.083634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.7.146.194:38557)
0619 06:44:35.094272 (+ 10638us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:35.094282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:35.095685 (+ 1403us) server_negotiation.cc:407] Connection header received
0619 06:44:35.095869 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:35.095877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:35.096107 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:35.096295 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:35.097107 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.098517 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.101490 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.102849 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.102991 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.103208 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:35.104430 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0619 06:44:35.104629 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10444,"thread_start_us":10330,"threads_started":1}
I20260619 06:44:35.105335 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.083127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:37023 (local address 127.0.0.1:49150)
0619 06:44:35.094459 (+ 11332us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:35.094520 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:35.094710 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:35.095927 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:35.095935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:35.095966 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:35.096385 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.096398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.098827 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.098832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.101278 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.101297 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.104588 (+ 3291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.104595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.104759 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.104792 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:44:35.104879 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:44:35.105006 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11017,"mutex_wait_us":127,"thread_start_us":11167,"threads_started":1}
I20260619 06:44:35.105463 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.083131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.7.146.193:37023)
0619 06:44:35.095067 (+ 11936us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:35.095081 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:35.095111 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:44:35.095212 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:35.095220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:35.095498 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:35.095700 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:35.096590 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.098665 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.102449 (+ 3784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.104350 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.104492 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.104752 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:35.104946 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 06:44:35.105113 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11807,"thread_start_us":11739,"threads_started":1}
I20260619 06:44:35.107757 26903 tablet_service.cc:1467] Processing CreateTablet for tablet 7c753e6fa4ea43dcb59e22dbaefb308a (DEFAULT_TABLE table=test-table [id=99839145580a4476ad49c1ffe6e95247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:35.108366 26642 tablet_service.cc:1467] Processing CreateTablet for tablet 7c753e6fa4ea43dcb59e22dbaefb308a (DEFAULT_TABLE table=test-table [id=99839145580a4476ad49c1ffe6e95247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:35.108717 26773 tablet_service.cc:1467] Processing CreateTablet for tablet 7c753e6fa4ea43dcb59e22dbaefb308a (DEFAULT_TABLE table=test-table [id=99839145580a4476ad49c1ffe6e95247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:35.109400 26903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c753e6fa4ea43dcb59e22dbaefb308a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:35.109652 26642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c753e6fa4ea43dcb59e22dbaefb308a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:35.109932 26773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c753e6fa4ea43dcb59e22dbaefb308a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:35.120895 26986 tablet_bootstrap.cc:492] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0: Bootstrap starting.
I20260619 06:44:35.124109 26987 tablet_bootstrap.cc:492] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c: Bootstrap starting.
I20260619 06:44:35.124379 26986 tablet_bootstrap.cc:654] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:35.125718 26986 log.cc:824] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:35.128475 26987 tablet_bootstrap.cc:654] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:35.129876 26987 log.cc:824] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:35.130007 26988 tablet_bootstrap.cc:492] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: Bootstrap starting.
I20260619 06:44:35.133239 26987 tablet_bootstrap.cc:492] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c: No bootstrap required, opened a new log
I20260619 06:44:35.133534 26987 ts_tablet_manager.cc:1397] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 06:44:35.134176 26986 tablet_bootstrap.cc:492] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0: No bootstrap required, opened a new log
I20260619 06:44:35.134369 26988 tablet_bootstrap.cc:654] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:35.134440 26986 ts_tablet_manager.cc:1397] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260619 06:44:35.135578 26988 log.cc:824] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:35.139043 26988 tablet_bootstrap.cc:492] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: No bootstrap required, opened a new log
I20260619 06:44:35.139302 26988 ts_tablet_manager.cc:1397] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260619 06:44:35.142207 26986 raft_consensus.cc:348] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972023942a57402e8ae2ec81d4659fc0" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37023 } } peers { permanent_uuid: "c310b86d31374f99a39d0545f945441c" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38557 } } peers { permanent_uuid: "0e134add5f964603bf46200d4b586ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 44137 } }
I20260619 06:44:35.142833 26986 raft_consensus.cc:374] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:35.142989 26986 raft_consensus.cc:729] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972023942a57402e8ae2ec81d4659fc0, State: Initialized, Role: FOLLOWER
I20260619 06:44:35.143579 26987 raft_consensus.cc:348] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972023942a57402e8ae2ec81d4659fc0" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37023 } } peers { permanent_uuid: "c310b86d31374f99a39d0545f945441c" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38557 } } peers { permanent_uuid: "0e134add5f964603bf46200d4b586ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 44137 } }
I20260619 06:44:35.143797 26986 consensus_queue.cc:260] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972023942a57402e8ae2ec81d4659fc0" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37023 } } peers { permanent_uuid: "c310b86d31374f99a39d0545f945441c" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38557 } } peers { permanent_uuid: "0e134add5f964603bf46200d4b586ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 44137 } }
I20260619 06:44:35.144096 26987 raft_consensus.cc:374] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:35.144212 26987 raft_consensus.cc:729] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c310b86d31374f99a39d0545f945441c, State: Initialized, Role: FOLLOWER
I20260619 06:44:35.144966 26987 consensus_queue.cc:260] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972023942a57402e8ae2ec81d4659fc0" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37023 } } peers { permanent_uuid: "c310b86d31374f99a39d0545f945441c" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38557 } } peers { permanent_uuid: "0e134add5f964603bf46200d4b586ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 44137 } }
I20260619 06:44:35.151206 26707 heartbeater.cc:503] Master 127.7.146.254:35453 was elected leader, sending a full tablet report...
I20260619 06:44:35.152324 26986 ts_tablet_manager.cc:1428] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0: Time spent starting tablet: real 0.018s user 0.008s sys 0.008s
I20260619 06:44:35.152804 26838 heartbeater.cc:503] Master 127.7.146.254:35453 was elected leader, sending a full tablet report...
I20260619 06:44:35.153146 26987 ts_tablet_manager.cc:1428] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
I20260619 06:44:35.155666 26988 raft_consensus.cc:348] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972023942a57402e8ae2ec81d4659fc0" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37023 } } peers { permanent_uuid: "c310b86d31374f99a39d0545f945441c" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38557 } } peers { permanent_uuid: "0e134add5f964603bf46200d4b586ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 44137 } }
I20260619 06:44:35.156214 26988 raft_consensus.cc:374] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:35.156371 26988 raft_consensus.cc:729] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e134add5f964603bf46200d4b586ee2, State: Initialized, Role: FOLLOWER
I20260619 06:44:35.157868 26988 consensus_queue.cc:260] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972023942a57402e8ae2ec81d4659fc0" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37023 } } peers { permanent_uuid: "c310b86d31374f99a39d0545f945441c" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38557 } } peers { permanent_uuid: "0e134add5f964603bf46200d4b586ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 44137 } }
I20260619 06:44:35.159071 26968 heartbeater.cc:503] Master 127.7.146.254:35453 was elected leader, sending a full tablet report...
I20260619 06:44:35.159663 26988 ts_tablet_manager.cc:1428] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: Time spent starting tablet: real 0.020s user 0.010s sys 0.004s
I20260619 06:44:35.223245 26993 raft_consensus.cc:482] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:35.223558 26993 raft_consensus.cc:504] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972023942a57402e8ae2ec81d4659fc0" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37023 } } peers { permanent_uuid: "c310b86d31374f99a39d0545f945441c" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38557 } } peers { permanent_uuid: "0e134add5f964603bf46200d4b586ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 44137 } }
I20260619 06:44:35.225201 26993 leader_election.cc:288] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 972023942a57402e8ae2ec81d4659fc0 (127.7.146.193:37023), 0e134add5f964603bf46200d4b586ee2 (127.7.146.195:44137)
I20260619 06:44:35.231856 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.225727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:34099 (local address 127.7.146.195:44137)
0619 06:44:35.225959 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:35.225965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:35.226680 (+ 715us) server_negotiation.cc:407] Connection header received
0619 06:44:35.226855 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:35.226858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:35.226915 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:35.226994 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:35.227566 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.228268 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.230140 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.230932 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.231091 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.231158 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:35.231496 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 06:44:35.231611 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 06:44:35.231873 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.225742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:36669 (local address 127.7.146.193:37023)
0619 06:44:35.225957 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:35.225964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:35.226736 (+ 772us) server_negotiation.cc:407] Connection header received
0619 06:44:35.226904 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:35.226908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:35.226979 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:35.227053 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:35.227672 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.228365 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.230153 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.230940 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.231083 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.231155 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:35.231494 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 06:44:35.231612 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:44:35.231986 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.225468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:44137 (local address 127.7.146.194:34099)
0619 06:44:35.226566 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:35.226581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:35.226730 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:35.227018 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:35.227022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:35.227043 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:35.227423 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.227436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.228390 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.228394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.229949 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.229961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.231124 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.231130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.231354 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.231381 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:44:35.231455 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:44:35.231585 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":23,"thread_start_us":710,"threads_started":1}
I20260619 06:44:35.232088 26995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.225468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:37023 (local address 127.7.146.194:36669)
0619 06:44:35.226621 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:35.226637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:35.226771 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:35.227082 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:35.227086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:35.227112 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:35.227517 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.227527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.228500 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.228503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.229977 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:35.229989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.231163 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.231171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:35.231338 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.231363 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:44:35.231440 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:44:35.231566 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":66,"thread_start_us":626,"threads_started":1}
I20260619 06:44:35.232980 26923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c753e6fa4ea43dcb59e22dbaefb308a" candidate_uuid: "c310b86d31374f99a39d0545f945441c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e134add5f964603bf46200d4b586ee2" is_pre_election: true
I20260619 06:44:35.233012 26662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c753e6fa4ea43dcb59e22dbaefb308a" candidate_uuid: "c310b86d31374f99a39d0545f945441c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972023942a57402e8ae2ec81d4659fc0" is_pre_election: true
I20260619 06:44:35.233450 26662 raft_consensus.cc:2454] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c310b86d31374f99a39d0545f945441c in term 0.
I20260619 06:44:35.233450 26923 raft_consensus.cc:2454] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c310b86d31374f99a39d0545f945441c in term 0.
I20260619 06:44:35.234040 26728 leader_election.cc:302] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 972023942a57402e8ae2ec81d4659fc0, c310b86d31374f99a39d0545f945441c; no voters:
I20260619 06:44:35.234390 26993 raft_consensus.cc:2781] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:44:35.234545 26993 raft_consensus.cc:482] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:44:35.234654 26993 raft_consensus.cc:3037] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:35.236878 26993 raft_consensus.cc:504] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972023942a57402e8ae2ec81d4659fc0" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37023 } } peers { permanent_uuid: "c310b86d31374f99a39d0545f945441c" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38557 } } peers { permanent_uuid: "0e134add5f964603bf46200d4b586ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 44137 } }
I20260619 06:44:35.237460 26993 leader_election.cc:288] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [CANDIDATE]: Term 1 election: Requested vote from peers 972023942a57402e8ae2ec81d4659fc0 (127.7.146.193:37023), 0e134add5f964603bf46200d4b586ee2 (127.7.146.195:44137)
I20260619 06:44:35.237942 26662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c753e6fa4ea43dcb59e22dbaefb308a" candidate_uuid: "c310b86d31374f99a39d0545f945441c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972023942a57402e8ae2ec81d4659fc0"
I20260619 06:44:35.238005 26923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c753e6fa4ea43dcb59e22dbaefb308a" candidate_uuid: "c310b86d31374f99a39d0545f945441c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e134add5f964603bf46200d4b586ee2"
I20260619 06:44:35.238162 26662 raft_consensus.cc:3037] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:35.238180 26923 raft_consensus.cc:3037] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:35.241356 26662 raft_consensus.cc:2454] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c310b86d31374f99a39d0545f945441c in term 1.
I20260619 06:44:35.241384 26923 raft_consensus.cc:2454] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c310b86d31374f99a39d0545f945441c in term 1.
I20260619 06:44:35.241946 26728 leader_election.cc:302] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e134add5f964603bf46200d4b586ee2, 972023942a57402e8ae2ec81d4659fc0, c310b86d31374f99a39d0545f945441c; no voters:
I20260619 06:44:35.242268 26993 raft_consensus.cc:2781] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:35.242694 26993 raft_consensus.cc:686] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 1 LEADER]: Becoming Leader. State: Replica: c310b86d31374f99a39d0545f945441c, State: Running, Role: LEADER
I20260619 06:44:35.243171 26993 consensus_queue.cc:237] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972023942a57402e8ae2ec81d4659fc0" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37023 } } peers { permanent_uuid: "c310b86d31374f99a39d0545f945441c" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38557 } } peers { permanent_uuid: "0e134add5f964603bf46200d4b586ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 44137 } }
I20260619 06:44:35.248308 26521 catalog_manager.cc:5495] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c reported cstate change: term changed from 0 to 1, leader changed from <none> to c310b86d31374f99a39d0545f945441c (127.7.146.194). New cstate: current_term: 1 leader_uuid: "c310b86d31374f99a39d0545f945441c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972023942a57402e8ae2ec81d4659fc0" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c310b86d31374f99a39d0545f945441c" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 38557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e134add5f964603bf46200d4b586ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 44137 } health_report { overall_health: UNKNOWN } } }
W20260619 06:44:35.277930 26969 tablet.cc:2369] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:44:35.294598 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.289559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.7.146.194:38557)
0619 06:44:35.289744 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:35.289749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:35.289767 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:44:35.289933 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:35.289937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:35.290011 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:35.290129 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:44:35.290440 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.291105 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.292498 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.292942 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.293039 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.293495 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:44:35.293998 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:44:35.294123 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:35.294248 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 06:44:35.294289 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:44:35.302913 26923 raft_consensus.cc:1261] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2 [term 1 FOLLOWER]: Refusing update from remote peer c310b86d31374f99a39d0545f945441c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:35.302911 26662 raft_consensus.cc:1261] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0 [term 1 FOLLOWER]: Refusing update from remote peer c310b86d31374f99a39d0545f945441c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:35.304105 26997 consensus_queue.cc:1035] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e134add5f964603bf46200d4b586ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:35.304476 26993 consensus_queue.cc:1035] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [LEADER]: Connected to new peer: Peer: permanent_uuid: "972023942a57402e8ae2ec81d4659fc0" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:35.317209 27002 mvcc.cc:205] Tried to move back new op lower bound from 7298463642830172160 to 7298463642599878656. Current Snapshot: MvccSnapshot[applied={T|T < 7298463642830172160}]
I20260619 06:44:35.329082 26520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:35.329332 26520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:35.333091 26520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7ffaabe6c9444e69b4ff841be31a278: Sending DeleteTablet for 3 replicas of tablet 7c753e6fa4ea43dcb59e22dbaefb308a
I20260619 06:44:35.334270 26773 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c753e6fa4ea43dcb59e22dbaefb308a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:35 UTC) from {username='slave'} at 127.0.0.1:38660
I20260619 06:44:35.334416 26642 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c753e6fa4ea43dcb59e22dbaefb308a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:35 UTC) from {username='slave'} at 127.0.0.1:49150
I20260619 06:44:35.335220 27011 tablet_replica.cc:330] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0: stopping tablet replica
I20260619 06:44:35.335305 26903 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c753e6fa4ea43dcb59e22dbaefb308a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:35 UTC) from {username='slave'} at 127.0.0.1:46942
I20260619 06:44:35.335651 27011 raft_consensus.cc:2229] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:35.336170 27011 raft_consensus.cc:2258] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:35.337612 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 26580
I20260619 06:44:35.343551 27012 tablet_replica.cc:330] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: stopping tablet replica
I20260619 06:44:35.343750 27010 tablet_replica.cc:330] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c: stopping tablet replica
I20260619 06:44:35.344163 27012 raft_consensus.cc:2229] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:35.344187 27010 raft_consensus.cc:2229] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:44:35.344825 27012 raft_consensus.cc:2258] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:35.344861 27010 raft_consensus.cc:2258] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:35.347019 27011 ts_tablet_manager.cc:1905] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:35.347023 27010 ts_tablet_manager.cc:1905] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:35.347404 27012 ts_tablet_manager.cc:1905] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:35.352344 27012 ts_tablet_manager.cc:1918] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:35.352489 27012 log.cc:1196] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/7c753e6fa4ea43dcb59e22dbaefb308a
I20260619 06:44:35.352928 27012 ts_tablet_manager.cc:1939] T 7c753e6fa4ea43dcb59e22dbaefb308a P 0e134add5f964603bf46200d4b586ee2: Deleting consensus metadata
I20260619 06:44:35.353796 27010 ts_tablet_manager.cc:1918] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:35.353899 27011 ts_tablet_manager.cc:1918] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:35.354063 27011 log.cc:1196] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/7c753e6fa4ea43dcb59e22dbaefb308a
I20260619 06:44:35.354144 27010 log.cc:1196] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/7c753e6fa4ea43dcb59e22dbaefb308a
I20260619 06:44:35.354667 27010 ts_tablet_manager.cc:1939] T 7c753e6fa4ea43dcb59e22dbaefb308a P c310b86d31374f99a39d0545f945441c: Deleting consensus metadata
I20260619 06:44:35.354657 26505 catalog_manager.cc:4841] TS 0e134add5f964603bf46200d4b586ee2 (127.7.146.195:44137): tablet 7c753e6fa4ea43dcb59e22dbaefb308a (table test-table [id=99839145580a4476ad49c1ffe6e95247]) successfully deleted
I20260619 06:44:35.356352 26506 catalog_manager.cc:4841] TS c310b86d31374f99a39d0545f945441c (127.7.146.194:38557): tablet 7c753e6fa4ea43dcb59e22dbaefb308a (table test-table [id=99839145580a4476ad49c1ffe6e95247]) successfully deleted
I20260619 06:44:35.357695 27011 ts_tablet_manager.cc:1939] T 7c753e6fa4ea43dcb59e22dbaefb308a P 972023942a57402e8ae2ec81d4659fc0: Deleting consensus metadata
I20260619 06:44:35.359203 26507 catalog_manager.cc:4841] TS 972023942a57402e8ae2ec81d4659fc0 (127.7.146.193:37023): tablet 7c753e6fa4ea43dcb59e22dbaefb308a (table test-table [id=99839145580a4476ad49c1ffe6e95247]) successfully deleted
I20260619 06:44:35.360636 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.338991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.7.146.193:37023)
0619 06:44:35.339184 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:35.339190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:35.345639 (+ 6449us) server_negotiation.cc:407] Connection header received
0619 06:44:35.346029 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:35.346035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:35.346135 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:35.346211 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:35.355000 (+ 8789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.355696 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.357181 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.357825 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.357957 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.359247 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:35.359257 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:35.359274 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:35.359860 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:35.359906 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:35.359926 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:35.360014 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:35.360168 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 06:44:35.360251 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:44:35.361217 26702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:35.479156 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 26580
I20260619 06:44:35.493286 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 26711
I20260619 06:44:35.500378 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.494151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.7.146.194:38557)
0619 06:44:35.494547 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:35.494551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:35.494566 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:44:35.494624 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:35.494630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:35.494759 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:35.494846 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:35.495290 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.496062 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.497347 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.497760 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.497879 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.498947 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:35.498959 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:35.498978 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:35.499637 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:35.499690 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:35.499710 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:35.499806 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:35.499939 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 06:44:35.500044 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
I20260619 06:44:35.501315 26833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:35.612310 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 26711
I20260619 06:44:35.626948 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 26841
I20260619 06:44:35.636931 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:35.627720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.7.146.195:44137)
0619 06:44:35.627889 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:35.627894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:35.627932 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:44:35.628173 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:35.628177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:35.628262 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:35.628344 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:35.628801 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.629526 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.630818 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:35.631199 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:35.631305 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:35.635612 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:35.635622 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:35.635636 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:35.636201 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:35.636253 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:35.636275 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:35.636374 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:35.636517 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 06:44:35.636655 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:44:35.637553 26963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:35.724071 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 26841
I20260619 06:44:35.736536 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 26490
I20260619 06:44:35.737603 26551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:35.845216 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 26490
2026-06-19T06:44:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2710 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:35 dist-test-slave-qfk4 krb5kdc[27020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:35 dist-test-slave-qfk4 krb5kdc[27020](info): set up 2 sockets
Jun 19 06:44:35 dist-test-slave-qfk4 krb5kdc[27020](info): commencing operation
krb5kdc: starting...
W20260619 06:44:36.860656 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:37 dist-test-slave-qfk4 krb5kdc[27020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:44:37.097121 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40129
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:40129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:37.419577 27033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:37.419895 27033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:37.420034 27033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:37.427325 27033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:37.427459 27033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:37.427528 27033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:37.427600 27033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:37.427657 27033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:37.437592 27033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:40129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:37.438876 27033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:37.440160 27033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:37.451659 27038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:37.451687 27039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:37.451741 27041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:37.452596 27033 server_base.cc:1034] running on GCE node
I20260619 06:44:37.453429 27033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:37.454900 27033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:37.456140 27033 hybrid_clock.cc:648] HybridClock initialized: now 1781851477456082 us; error 32 us; skew 500 ppm
I20260619 06:44:37.456516 27033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:37 dist-test-slave-qfk4 krb5kdc[27020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851477, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:37.459721 27033 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:44:37.461411 27033 webserver.cc:460] Webserver started at http://127.7.146.254:33629/ using document root <none> and password file <none>
I20260619 06:44:37.462116 27033 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:37.462237 27033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:37.462540 27033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:37.465662 27033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "b208d0b4db7741e2b23883754db2b949"
format_stamp: "Formatted at 2026-06-19 06:44:37 on dist-test-slave-qfk4"
server_key: "3c92f5f6ca0a0419f03a949909c4db9d"
server_key_iv: "f0bcfc995a7ae722656ec19da7f3305a"
server_key_version: "encryptionkey@0"
I20260619 06:44:37.466409 27033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "b208d0b4db7741e2b23883754db2b949"
format_stamp: "Formatted at 2026-06-19 06:44:37 on dist-test-slave-qfk4"
server_key: "3c92f5f6ca0a0419f03a949909c4db9d"
server_key_iv: "f0bcfc995a7ae722656ec19da7f3305a"
server_key_version: "encryptionkey@0"
I20260619 06:44:37.471644 27033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:44:37.475139 27048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:37.476496 27033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:44:37.476709 27033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "b208d0b4db7741e2b23883754db2b949"
format_stamp: "Formatted at 2026-06-19 06:44:37 on dist-test-slave-qfk4"
server_key: "3c92f5f6ca0a0419f03a949909c4db9d"
server_key_iv: "f0bcfc995a7ae722656ec19da7f3305a"
server_key_version: "encryptionkey@0"
I20260619 06:44:37.476893 27033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:37.495136 27033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:37.499377 27033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:37.499614 27033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:37.516825 27033 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:40129
I20260619 06:44:37.516839 27099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:40129 every 8 connection(s)
I20260619 06:44:37.518178 27033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:37.519331 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 27033
I20260619 06:44:37.519521 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:37.520035 7755 external_mini_cluster.cc:1428] Setting key 16b8dfdce0202e33da10beb323eef1b7
I20260619 06:44:37.522675 27100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:44:37 dist-test-slave-qfk4 krb5kdc[27020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:37.535651 27100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949: Bootstrap starting.
I20260619 06:44:37.538965 27100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:37.540139 27100 log.cc:824] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:37.542466 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:37.521859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.7.146.254:40129)
0619 06:44:37.522578 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:37.522590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:37.522631 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:44:37.524248 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:37.524268 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:37.524692 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:37.525030 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:37.526785 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:37.528159 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:37.529943 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:37.530601 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:37.530724 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:37.535448 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:37.535457 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:37.535473 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:37.535511 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:37.538374 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:37.539189 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:37.539196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:37.539202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:37.539375 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:37.539750 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:37.539753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:37.539754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:37.540264 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:37.540488 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:37.541059 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 06:44:37.541498 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":321,"threads_started":1}
I20260619 06:44:37.543078 27100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949: No bootstrap required, opened a new log
I20260619 06:44:37.550704 27100 raft_consensus.cc:348] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b208d0b4db7741e2b23883754db2b949" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40129 } }
I20260619 06:44:37.551084 27100 raft_consensus.cc:374] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:37.551182 27100 raft_consensus.cc:729] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b208d0b4db7741e2b23883754db2b949, State: Initialized, Role: FOLLOWER
I20260619 06:44:37.551723 27100 consensus_queue.cc:260] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b208d0b4db7741e2b23883754db2b949" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40129 } }
I20260619 06:44:37.551937 27100 raft_consensus.cc:388] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:37.552059 27100 raft_consensus.cc:482] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:37.552223 27100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:37.554809 27100 raft_consensus.cc:504] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b208d0b4db7741e2b23883754db2b949" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40129 } }
I20260619 06:44:37.555378 27100 leader_election.cc:302] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b208d0b4db7741e2b23883754db2b949; no voters:
I20260619 06:44:37.556003 27100 leader_election.cc:288] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:37.556128 27105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:37.557399 27105 raft_consensus.cc:686] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [term 1 LEADER]: Becoming Leader. State: Replica: b208d0b4db7741e2b23883754db2b949, State: Running, Role: LEADER
I20260619 06:44:37.557955 27105 consensus_queue.cc:237] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b208d0b4db7741e2b23883754db2b949" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40129 } }
I20260619 06:44:37.558332 27100 sys_catalog.cc:564] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:37.560755 27107 sys_catalog.cc:455] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b208d0b4db7741e2b23883754db2b949. Latest consensus state: current_term: 1 leader_uuid: "b208d0b4db7741e2b23883754db2b949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b208d0b4db7741e2b23883754db2b949" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40129 } } }
I20260619 06:44:37.561206 27107 sys_catalog.cc:458] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:37.561615 27106 sys_catalog.cc:455] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b208d0b4db7741e2b23883754db2b949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b208d0b4db7741e2b23883754db2b949" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40129 } } }
I20260619 06:44:37.561902 27106 sys_catalog.cc:458] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:37.564066 27114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:37.568409 27114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:37.577386 27114 catalog_manager.cc:1347] Generated new cluster ID: 5ae9cb5a53f14d45be5ee985843ed0f8
I20260619 06:44:37.577529 27114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:37.592262 27114 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:37.593778 27114 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:37.603246 27114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949: Generated new TSK 0
I20260619 06:44:37.604137 27114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.146.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
Entry for principal kudu/127.7.146.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab.
I20260619 06:44:37.673295 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40129
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:38.000543 27126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:38.000887 27126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:38.001019 27126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:38.008280 27126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:38.008402 27126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:38.008554 27126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:44:38.018515 27126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.193.keytab
--principal=kudu/127.7.146.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:38.019845 27126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:38.021201 27126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:38.034441 27134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:38.034596 27131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:38.034415 27132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:38.035368 27126 server_base.cc:1034] running on GCE node
I20260619 06:44:38.036005 27126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:38.037040 27126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:38.038280 27126 hybrid_clock.cc:648] HybridClock initialized: now 1781851478038189 us; error 55 us; skew 500 ppm
I20260619 06:44:38.038673 27126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:38 dist-test-slave-qfk4 krb5kdc[27020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851478, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:38.042315 27126 init.cc:377] Logged in from keytab as kudu/127.7.146.193@KRBTEST.COM (short username kudu)
I20260619 06:44:38.045809 27126 webserver.cc:460] Webserver started at http://127.7.146.193:39145/ using document root <none> and password file <none>
I20260619 06:44:38.046545 27126 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:38.046669 27126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:38.046983 27126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:38.049973 27126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "abb8479a4e41448aaf53e2140f9f1f09"
format_stamp: "Formatted at 2026-06-19 06:44:38 on dist-test-slave-qfk4"
server_key: "a87b046721bc557a7a803d5ceedde417"
server_key_iv: "76c4ddcda793d657a47a2fc6754be1ff"
server_key_version: "encryptionkey@0"
I20260619 06:44:38.050734 27126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "abb8479a4e41448aaf53e2140f9f1f09"
format_stamp: "Formatted at 2026-06-19 06:44:38 on dist-test-slave-qfk4"
server_key: "a87b046721bc557a7a803d5ceedde417"
server_key_iv: "76c4ddcda793d657a47a2fc6754be1ff"
server_key_version: "encryptionkey@0"
I20260619 06:44:38.055788 27126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 06:44:38.059530 27141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:38.061023 27126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:44:38.061173 27126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "abb8479a4e41448aaf53e2140f9f1f09"
format_stamp: "Formatted at 2026-06-19 06:44:38 on dist-test-slave-qfk4"
server_key: "a87b046721bc557a7a803d5ceedde417"
server_key_iv: "76c4ddcda793d657a47a2fc6754be1ff"
server_key_version: "encryptionkey@0"
I20260619 06:44:38.061349 27126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:38.078552 27126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:38.082962 27126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:38.083223 27126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:38.084192 27126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:38.085613 27126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:38.085695 27126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:38.085815 27126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:38.085865 27126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:38.115468 27126 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:35551
I20260619 06:44:38.115504 27253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:35551 every 8 connection(s)
I20260619 06:44:38.116780 27126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:44:38.119920 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 27126
I20260619 06:44:38.120111 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:44:38.120714 7755 external_mini_cluster.cc:1428] Setting key 82512e4d0b967f5050aa1776c4f7ce3d
Jun 19 06:44:38 dist-test-slave-qfk4 krb5kdc[27020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851478, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.193@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:38.138340 27256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:38.119231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:48003 (local address 127.7.146.254:40129)
0619 06:44:38.119687 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:38.119693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:38.120847 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 06:44:38.123120 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:38.123124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:38.123199 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:38.123296 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:38.125375 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:38.126097 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:38.128108 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:38.128509 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:38.128668 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:38.132478 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:38.132486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:38.132490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:38.132533 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:38.135012 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:38.135864 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:38.135869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:38.135871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:38.135999 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:38.136490 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:38.136494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:38.136497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:38.136846 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:38.137016 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:38.137841 (+ 825us) server_negotiation.cc:299] Negotiation successful
0619 06:44:38.138068 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":209,"threads_started":1}
I20260619 06:44:38.139884 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:38.119842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40129 (local address 127.7.146.193:48003)
0619 06:44:38.120667 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:38.120728 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:38.122794 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:38.123598 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:38.123610 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:38.124359 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:38.125176 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:38.125199 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:38.126244 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:38.126249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:38.127956 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:38.127971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:38.128784 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:38.128791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:38.129074 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:38.129759 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:44:38.129791 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:38.132229 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:38.135156 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:38.135165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:38.135182 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:38.135715 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:38.136102 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:38.136107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:38.136110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:38.136347 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:38.136961 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:38.136968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:38.137427 (+ 459us) client_negotiation.cc:769] Sending connection context
0619 06:44:38.137733 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 06:44:38.138319 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":296,"threads_started":1}
I20260619 06:44:38.141508 27254 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40129
I20260619 06:44:38.141845 27254 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:38.142719 27254 heartbeater.cc:511] Master 127.7.146.254:40129 requested a full tablet report, sending...
I20260619 06:44:38.145098 27065 ts_manager.cc:194] Registered new tserver with Master: abb8479a4e41448aaf53e2140f9f1f09 (127.7.146.193:35551)
WARNING: no policy specified for kudu/127.7.146.194@KRBTEST.COM; defaulting to no policy
I20260619 06:44:38.146605 27065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.193@KRBTEST.COM'} at 127.7.146.193:48003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
Entry for principal kudu/127.7.146.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab.
I20260619 06:44:38.176541 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40129
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:38.496104 27260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:38.496402 27260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:38.496500 27260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:38.503895 27260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:38.504019 27260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:38.504143 27260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:44:38.514020 27260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.194.keytab
--principal=kudu/127.7.146.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:38.515237 27260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:38.516729 27260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:38.530989 27265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:38.531538 27266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:38.531694 27268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:38.532073 27260 server_base.cc:1034] running on GCE node
I20260619 06:44:38.532814 27260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:38.533944 27260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:38.535183 27260 hybrid_clock.cc:648] HybridClock initialized: now 1781851478535135 us; error 36 us; skew 500 ppm
I20260619 06:44:38.535497 27260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:38 dist-test-slave-qfk4 krb5kdc[27020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851478, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:38.538642 27260 init.cc:377] Logged in from keytab as kudu/127.7.146.194@KRBTEST.COM (short username kudu)
I20260619 06:44:38.540167 27260 webserver.cc:460] Webserver started at http://127.7.146.194:32957/ using document root <none> and password file <none>
I20260619 06:44:38.540906 27260 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:38.541026 27260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:38.541316 27260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:38.544152 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "a249f39cc6ca43099575d570300ce650"
format_stamp: "Formatted at 2026-06-19 06:44:38 on dist-test-slave-qfk4"
server_key: "b5c0f1934cd710e2f147d65742cf2179"
server_key_iv: "676d031d850f2330135cc128460a0971"
server_key_version: "encryptionkey@0"
I20260619 06:44:38.544938 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "a249f39cc6ca43099575d570300ce650"
format_stamp: "Formatted at 2026-06-19 06:44:38 on dist-test-slave-qfk4"
server_key: "b5c0f1934cd710e2f147d65742cf2179"
server_key_iv: "676d031d850f2330135cc128460a0971"
server_key_version: "encryptionkey@0"
I20260619 06:44:38.550083 27260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 06:44:38.553547 27275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:38.554908 27260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:44:38.555054 27260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "a249f39cc6ca43099575d570300ce650"
format_stamp: "Formatted at 2026-06-19 06:44:38 on dist-test-slave-qfk4"
server_key: "b5c0f1934cd710e2f147d65742cf2179"
server_key_iv: "676d031d850f2330135cc128460a0971"
server_key_version: "encryptionkey@0"
I20260619 06:44:38.555230 27260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:38.575489 27260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:38.579875 27260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:38.580150 27260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:38.581154 27260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:38.582511 27260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:38.582585 27260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:38.582705 27260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:38.582756 27260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:38.612353 27260 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:35611
I20260619 06:44:38.612398 27387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:35611 every 8 connection(s)
I20260619 06:44:38.613611 27260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:44:38.619575 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 27260
I20260619 06:44:38.619758 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:44:38.620293 7755 external_mini_cluster.cc:1428] Setting key 9feadbb966fd3ac8db6dfc7d68e50b53
Jun 19 06:44:38 dist-test-slave-qfk4 krb5kdc[27020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851478, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.194@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:38.634896 27256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:38.616171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:42069 (local address 127.7.146.254:40129)
0619 06:44:38.616338 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:38.616343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:38.617497 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 06:44:38.619239 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:38.619243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:38.619316 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:38.619400 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:38.621364 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:38.622378 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:38.624540 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:38.625061 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:38.625159 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:38.629049 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:38.629053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:38.629057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:38.629089 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:38.631952 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:38.632678 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:38.632683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:38.632687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:38.632829 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:38.633249 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:38.633253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:38.633256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:38.633563 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:38.633699 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:38.634375 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 06:44:38.634593 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:44:38.635782 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:38.616627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40129 (local address 127.7.146.194:42069)
0619 06:44:38.617308 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:38.617351 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:38.618980 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:38.619679 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:38.619692 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:38.620405 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:38.621166 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:38.621191 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:38.622527 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:38.622532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:38.624371 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:38.624399 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:38.625273 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:38.625279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:38.625550 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:38.626225 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:44:38.626257 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:38.628808 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:38.632091 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:38.632098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:38.632110 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:38.632546 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:38.632954 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:38.632958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:38.632961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:38.633149 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:38.633688 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:38.633694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:38.634033 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 06:44:38.634256 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 06:44:38.634677 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":260,"threads_started":1}
I20260619 06:44:38.637233 27388 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40129
I20260619 06:44:38.637552 27388 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:38.638389 27388 heartbeater.cc:511] Master 127.7.146.254:40129 requested a full tablet report, sending...
I20260619 06:44:38.639978 27065 ts_manager.cc:194] Registered new tserver with Master: a249f39cc6ca43099575d570300ce650 (127.7.146.194:35611)
I20260619 06:44:38.640961 27065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.194@KRBTEST.COM'} at 127.7.146.194:42069
WARNING: no policy specified for kudu/127.7.146.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
Entry for principal kudu/127.7.146.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab.
I20260619 06:44:38.674405 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40129
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:38.993201 27393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:38.993525 27393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:38.993649 27393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:39.001008 27393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:39.001149 27393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:39.001307 27393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:44:39.011534 27393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.195.keytab
--principal=kudu/127.7.146.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:40129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:39.012897 27393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:39.014225 27393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:39.027007 27398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:39.027029 27401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:39.027007 27399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:39.027729 27393 server_base.cc:1034] running on GCE node
I20260619 06:44:39.028388 27393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:39.029376 27393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:39.030589 27393 hybrid_clock.cc:648] HybridClock initialized: now 1781851479030539 us; error 43 us; skew 500 ppm
I20260619 06:44:39.030975 27393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:39 dist-test-slave-qfk4 krb5kdc[27020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:39.034040 27393 init.cc:377] Logged in from keytab as kudu/127.7.146.195@KRBTEST.COM (short username kudu)
I20260619 06:44:39.035478 27393 webserver.cc:460] Webserver started at http://127.7.146.195:46611/ using document root <none> and password file <none>
I20260619 06:44:39.036164 27393 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:39.036280 27393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:39.036615 27393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:39.039731 27393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "e7033baf2ef04118bf0ac6119f6d5ee2"
format_stamp: "Formatted at 2026-06-19 06:44:39 on dist-test-slave-qfk4"
server_key: "fd84fb8a8efae887ae27d431bcbcb806"
server_key_iv: "a2cd848b2227515e5e36ee1b9eb4cec9"
server_key_version: "encryptionkey@0"
I20260619 06:44:39.040650 27393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "e7033baf2ef04118bf0ac6119f6d5ee2"
format_stamp: "Formatted at 2026-06-19 06:44:39 on dist-test-slave-qfk4"
server_key: "fd84fb8a8efae887ae27d431bcbcb806"
server_key_iv: "a2cd848b2227515e5e36ee1b9eb4cec9"
server_key_version: "encryptionkey@0"
I20260619 06:44:39.046327 27393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 06:44:39.050161 27408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:39.051627 27393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:44:39.051800 27393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "e7033baf2ef04118bf0ac6119f6d5ee2"
format_stamp: "Formatted at 2026-06-19 06:44:39 on dist-test-slave-qfk4"
server_key: "fd84fb8a8efae887ae27d431bcbcb806"
server_key_iv: "a2cd848b2227515e5e36ee1b9eb4cec9"
server_key_version: "encryptionkey@0"
I20260619 06:44:39.052007 27393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:39.068498 27393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:39.072299 27393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:39.072556 27393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:39.073523 27393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:39.074862 27393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:39.074939 27393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:39.075062 27393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:39.075114 27393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:39.104741 27393 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:41859
I20260619 06:44:39.104780 27520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:41859 every 8 connection(s)
I20260619 06:44:39.106000 27393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:44:39.107518 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 27393
I20260619 06:44:39.107707 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:44:39.108201 7755 external_mini_cluster.cc:1428] Setting key d7aed1a0a4d0c2ad840dfe1b9696922c
Jun 19 06:44:39 dist-test-slave-qfk4 krb5kdc[27020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.195@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:44:39.126287 27256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.108690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:51001 (local address 127.7.146.254:40129)
0619 06:44:39.108871 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:39.108877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:39.110329 (+ 1452us) server_negotiation.cc:407] Connection header received
0619 06:44:39.112199 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:39.112204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:39.112288 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:39.112370 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:39.113903 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.114680 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.116237 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.116695 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.116797 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.120169 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:39.120174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:39.120177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:39.120210 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:39.123260 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:39.123998 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:39.124002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:39.124005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:39.124125 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:39.124540 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:39.124545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:39.124548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:39.124861 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:39.124999 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:39.125697 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 06:44:39.125926 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:44:39.127254 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.109361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:40129 (local address 127.7.146.195:51001)
0619 06:44:39.110191 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:39.110228 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:39.111955 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:39.112637 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:39.112646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:39.113172 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:39.113742 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.113759 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.114824 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.114828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.116106 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.116118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.116905 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.116912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.117140 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.117745 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:44:39.117768 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:39.119954 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:39.123410 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:39.123417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:39.123431 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:39.123870 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:39.124232 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:44:39.124235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:44:39.124238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:44:39.124434 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:44:39.124985 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:39.124994 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:39.125337 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 06:44:39.125592 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 06:44:39.126140 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":284,"threads_started":1}
I20260619 06:44:39.128602 27521 heartbeater.cc:347] Connected to a master server at 127.7.146.254:40129
I20260619 06:44:39.128885 27521 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:39.129575 27521 heartbeater.cc:511] Master 127.7.146.254:40129 requested a full tablet report, sending...
I20260619 06:44:39.131021 27065 ts_manager.cc:194] Registered new tserver with Master: e7033baf2ef04118bf0ac6119f6d5ee2 (127.7.146.195:41859)
I20260619 06:44:39.131815 27065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.146.195@KRBTEST.COM'} at 127.7.146.195:51001
I20260619 06:44:39.135488 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:44:39.149797 27254 heartbeater.cc:503] Master 127.7.146.254:40129 was elected leader, sending a full tablet report...
I20260619 06:44:39.151357 27256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.139328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.7.146.254:40129)
0619 06:44:39.139510 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:39.139517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:39.139975 (+ 458us) server_negotiation.cc:407] Connection header received
0619 06:44:39.140126 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:39.140130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:39.140204 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:39.140276 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:39.141606 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.142338 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.143581 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.144008 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.144103 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.146286 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:39.146291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:39.146294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:39.146326 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:39.148788 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:39.149384 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:39.149389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:39.149393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:39.149550 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:39.149975 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:39.149979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:39.149980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:39.150303 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:39.150543 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:39.150877 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 06:44:39.151087 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:44:39.155198 27065 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:44:39.162235 27065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:44:39.188341 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.174373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.7.146.193:35551)
0619 06:44:39.176616 (+ 2243us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:39.176625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:39.176656 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:44:39.176756 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:39.176763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:39.177026 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:39.177224 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:39.178005 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.181192 (+ 3187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.184040 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.185722 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.187720 (+ 1998us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.187901 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:39.187960 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 06:44:39.188089 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2033,"thread_start_us":1880,"threads_started":1}
I20260619 06:44:39.188828 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.174190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35551 (local address 127.0.0.1:38776)
0619 06:44:39.175751 (+ 1561us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:39.175797 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:39.175956 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:39.177258 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:39.177266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:39.177306 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:39.177737 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.177752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.181361 (+ 3609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.181367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.183860 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.183876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.185987 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.185993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.186234 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.186296 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 06:44:39.186424 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 06:44:39.186657 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"thread_start_us":2891,"threads_started":1}
I20260619 06:44:39.189420 27533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.174431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:35611 (local address 127.0.0.1:57006)
0619 06:44:39.177352 (+ 2921us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:39.177370 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:39.177559 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:39.178471 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:39.178477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:39.178504 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:39.178949 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.178961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.181182 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.181189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.185269 (+ 4080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.185282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.188645 (+ 3363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.188656 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.188805 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.188832 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:44:39.188919 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:44:39.189056 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2795,"thread_start_us":925,"threads_started":1}
I20260619 06:44:39.189479 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.174811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.7.146.194:35611)
0619 06:44:39.176961 (+ 2150us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:39.176968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:39.177455 (+ 487us) server_negotiation.cc:407] Connection header received
0619 06:44:39.177758 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:39.177765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:39.178016 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:39.178187 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:39.179135 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.180920 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.185456 (+ 4536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.187010 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.187114 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.187327 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:39.188985 (+ 1658us) server_negotiation.cc:299] Negotiation successful
0619 06:44:39.189157 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1986,"thread_start_us":1886,"threads_started":1}
I20260619 06:44:39.193327 27189 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1f388ddb974064ad78a180142ab593 (DEFAULT_TABLE table=test-table [id=5b046d9e5d4f41e988f00e3ad568a198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:39.193710 27323 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1f388ddb974064ad78a180142ab593 (DEFAULT_TABLE table=test-table [id=5b046d9e5d4f41e988f00e3ad568a198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:39.194960 27189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1f388ddb974064ad78a180142ab593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:39.194998 27323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1f388ddb974064ad78a180142ab593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:39.195772 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.177804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.7.146.195:41859)
0619 06:44:39.187221 (+ 9417us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:39.187230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:39.189349 (+ 2119us) server_negotiation.cc:407] Connection header received
0619 06:44:39.189461 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:39.189469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:39.189816 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:39.190064 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:39.190633 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.191763 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.193779 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.194787 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.194904 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.195129 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:39.195321 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 06:44:39.195480 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9226,"thread_start_us":9070,"threads_started":1}
I20260619 06:44:39.195760 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.177493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:41859 (local address 127.0.0.1:47038)
0619 06:44:39.189048 (+ 11555us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:39.189066 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:39.189193 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:39.190071 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:39.190076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:39.190103 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:39.190442 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.190455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.191896 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.191901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.193636 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.193648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.194975 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.194981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.195147 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.195179 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:44:39.195261 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:44:39.195397 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10026,"thread_start_us":9759,"threads_started":1}
I20260619 06:44:39.199147 27456 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1f388ddb974064ad78a180142ab593 (DEFAULT_TABLE table=test-table [id=5b046d9e5d4f41e988f00e3ad568a198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:39.200430 27456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1f388ddb974064ad78a180142ab593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:39.209806 27539 tablet_bootstrap.cc:492] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: Bootstrap starting.
I20260619 06:44:39.210600 27540 tablet_bootstrap.cc:492] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: Bootstrap starting.
I20260619 06:44:39.213004 27541 tablet_bootstrap.cc:492] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: Bootstrap starting.
I20260619 06:44:39.214249 27540 tablet_bootstrap.cc:654] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:39.215816 27540 log.cc:824] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:39.216135 27539 tablet_bootstrap.cc:654] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:39.216722 27541 tablet_bootstrap.cc:654] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:39.217394 27539 log.cc:824] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:39.217892 27541 log.cc:824] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:39.220683 27539 tablet_bootstrap.cc:492] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: No bootstrap required, opened a new log
I20260619 06:44:39.220968 27539 ts_tablet_manager.cc:1397] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260619 06:44:39.221095 27540 tablet_bootstrap.cc:492] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: No bootstrap required, opened a new log
I20260619 06:44:39.221393 27540 ts_tablet_manager.cc:1397] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260619 06:44:39.221683 27541 tablet_bootstrap.cc:492] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: No bootstrap required, opened a new log
I20260619 06:44:39.221972 27541 ts_tablet_manager.cc:1397] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 06:44:39.228530 27539 raft_consensus.cc:348] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb8479a4e41448aaf53e2140f9f1f09" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35551 } } peers { permanent_uuid: "a249f39cc6ca43099575d570300ce650" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35611 } } peers { permanent_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41859 } }
I20260619 06:44:39.228983 27539 raft_consensus.cc:374] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:39.229094 27539 raft_consensus.cc:729] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a249f39cc6ca43099575d570300ce650, State: Initialized, Role: FOLLOWER
I20260619 06:44:39.229662 27539 consensus_queue.cc:260] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb8479a4e41448aaf53e2140f9f1f09" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35551 } } peers { permanent_uuid: "a249f39cc6ca43099575d570300ce650" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35611 } } peers { permanent_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41859 } }
I20260619 06:44:39.230711 27388 heartbeater.cc:503] Master 127.7.146.254:40129 was elected leader, sending a full tablet report...
I20260619 06:44:39.231457 27539 ts_tablet_manager.cc:1428] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260619 06:44:39.231756 27540 raft_consensus.cc:348] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb8479a4e41448aaf53e2140f9f1f09" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35551 } } peers { permanent_uuid: "a249f39cc6ca43099575d570300ce650" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35611 } } peers { permanent_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41859 } }
I20260619 06:44:39.232201 27540 raft_consensus.cc:374] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:39.231986 27541 raft_consensus.cc:348] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb8479a4e41448aaf53e2140f9f1f09" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35551 } } peers { permanent_uuid: "a249f39cc6ca43099575d570300ce650" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35611 } } peers { permanent_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41859 } }
I20260619 06:44:39.232407 27541 raft_consensus.cc:374] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:39.232393 27540 raft_consensus.cc:729] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb8479a4e41448aaf53e2140f9f1f09, State: Initialized, Role: FOLLOWER
I20260619 06:44:39.232506 27541 raft_consensus.cc:729] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7033baf2ef04118bf0ac6119f6d5ee2, State: Initialized, Role: FOLLOWER
I20260619 06:44:39.233040 27540 consensus_queue.cc:260] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb8479a4e41448aaf53e2140f9f1f09" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35551 } } peers { permanent_uuid: "a249f39cc6ca43099575d570300ce650" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35611 } } peers { permanent_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41859 } }
I20260619 06:44:39.233083 27541 consensus_queue.cc:260] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb8479a4e41448aaf53e2140f9f1f09" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35551 } } peers { permanent_uuid: "a249f39cc6ca43099575d570300ce650" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35611 } } peers { permanent_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41859 } }
I20260619 06:44:39.234808 27541 ts_tablet_manager.cc:1428] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260619 06:44:39.235102 27521 heartbeater.cc:503] Master 127.7.146.254:40129 was elected leader, sending a full tablet report...
I20260619 06:44:39.235165 27540 ts_tablet_manager.cc:1428] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260619 06:44:39.357517 27522 tablet.cc:2369] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:44:39.365561 27389 tablet.cc:2369] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:44:39.368970 27255 tablet.cc:2369] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:44:39.421464 27547 raft_consensus.cc:482] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:39.421797 27547 raft_consensus.cc:504] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb8479a4e41448aaf53e2140f9f1f09" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35551 } } peers { permanent_uuid: "a249f39cc6ca43099575d570300ce650" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35611 } } peers { permanent_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41859 } }
I20260619 06:44:39.423615 27547 leader_election.cc:288] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abb8479a4e41448aaf53e2140f9f1f09 (127.7.146.193:35551), a249f39cc6ca43099575d570300ce650 (127.7.146.194:35611)
I20260619 06:44:39.430055 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.424091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:54437 (local address 127.7.146.193:35551)
0619 06:44:39.424292 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:39.424298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:39.424320 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:44:39.424580 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:39.424587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:39.424669 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:39.424769 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:39.425182 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.426127 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.428015 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.429224 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.429339 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.429392 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:39.429737 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 06:44:39.429837 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:44:39.430176 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.424080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:35551 (local address 127.7.146.195:54437)
0619 06:44:39.424247 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:39.424266 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:39.424408 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:39.424803 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:39.424808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:39.424840 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:39.425027 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.425036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.426260 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.426265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.427849 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.427861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.429364 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.429370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.429577 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.429613 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:44:39.429719 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:44:39.429861 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 06:44:39.431083 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.424080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:35611 (local address 127.7.146.195:33311)
0619 06:44:39.425006 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:39.425019 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:39.425120 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:39.425396 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:39.425400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:39.425448 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:39.425767 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.425778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.427164 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.427167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.428910 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:39.428926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.430428 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.430435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:39.430597 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.430621 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:44:39.430687 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 06:44:39.430772 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":173,"thread_start_us":416,"threads_started":1}
I20260619 06:44:39.431118 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.424164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:33311 (local address 127.7.146.194:35611)
0619 06:44:39.424335 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:39.424341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:39.425106 (+ 765us) server_negotiation.cc:407] Connection header received
0619 06:44:39.425234 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:39.425238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:39.425296 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:39.425363 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:39.425962 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.427013 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.429069 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.430185 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.430332 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.430398 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:39.430766 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 06:44:39.430944 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:44:39.431460 27209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f388ddb974064ad78a180142ab593" candidate_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb8479a4e41448aaf53e2140f9f1f09" is_pre_election: true
I20260619 06:44:39.431993 27209 raft_consensus.cc:2454] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7033baf2ef04118bf0ac6119f6d5ee2 in term 0.
I20260619 06:44:39.432108 27343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f388ddb974064ad78a180142ab593" candidate_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a249f39cc6ca43099575d570300ce650" is_pre_election: true
I20260619 06:44:39.432693 27343 raft_consensus.cc:2454] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7033baf2ef04118bf0ac6119f6d5ee2 in term 0.
I20260619 06:44:39.432708 27411 leader_election.cc:302] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abb8479a4e41448aaf53e2140f9f1f09, e7033baf2ef04118bf0ac6119f6d5ee2; no voters:
I20260619 06:44:39.433060 27547 raft_consensus.cc:2781] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:44:39.433236 27547 raft_consensus.cc:482] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:44:39.433346 27547 raft_consensus.cc:3037] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:39.435520 27547 raft_consensus.cc:504] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb8479a4e41448aaf53e2140f9f1f09" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35551 } } peers { permanent_uuid: "a249f39cc6ca43099575d570300ce650" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35611 } } peers { permanent_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41859 } }
I20260619 06:44:39.436100 27547 leader_election.cc:288] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [CANDIDATE]: Term 1 election: Requested vote from peers abb8479a4e41448aaf53e2140f9f1f09 (127.7.146.193:35551), a249f39cc6ca43099575d570300ce650 (127.7.146.194:35611)
I20260619 06:44:39.436465 27209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f388ddb974064ad78a180142ab593" candidate_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb8479a4e41448aaf53e2140f9f1f09"
I20260619 06:44:39.436532 27343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f388ddb974064ad78a180142ab593" candidate_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a249f39cc6ca43099575d570300ce650"
I20260619 06:44:39.436690 27209 raft_consensus.cc:3037] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:39.436764 27343 raft_consensus.cc:3037] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:39.438958 27343 raft_consensus.cc:2454] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7033baf2ef04118bf0ac6119f6d5ee2 in term 1.
I20260619 06:44:39.438970 27209 raft_consensus.cc:2454] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7033baf2ef04118bf0ac6119f6d5ee2 in term 1.
I20260619 06:44:39.439329 27409 leader_election.cc:302] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a249f39cc6ca43099575d570300ce650, e7033baf2ef04118bf0ac6119f6d5ee2; no voters:
I20260619 06:44:39.439602 27547 raft_consensus.cc:2781] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:39.440035 27547 raft_consensus.cc:686] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 1 LEADER]: Becoming Leader. State: Replica: e7033baf2ef04118bf0ac6119f6d5ee2, State: Running, Role: LEADER
I20260619 06:44:39.440486 27547 consensus_queue.cc:237] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb8479a4e41448aaf53e2140f9f1f09" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35551 } } peers { permanent_uuid: "a249f39cc6ca43099575d570300ce650" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35611 } } peers { permanent_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41859 } }
I20260619 06:44:39.445191 27065 catalog_manager.cc:5495] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7033baf2ef04118bf0ac6119f6d5ee2 (127.7.146.195). New cstate: current_term: 1 leader_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb8479a4e41448aaf53e2140f9f1f09" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a249f39cc6ca43099575d570300ce650" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7033baf2ef04118bf0ac6119f6d5ee2" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 41859 } health_report { overall_health: HEALTHY } } }
I20260619 06:44:39.481413 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.476074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.7.146.195:41859)
0619 06:44:39.476270 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:39.476276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:39.476346 (+ 70us) server_negotiation.cc:407] Connection header received
0619 06:44:39.476618 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:39.476622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:39.476702 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:39.476840 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:44:39.477205 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.477948 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.479293 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.479686 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.479794 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.480279 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:44:39.480875 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:44:39.481014 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:39.481088 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 06:44:39.481169 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:44:39.489482 27343 raft_consensus.cc:1261] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650 [term 1 FOLLOWER]: Refusing update from remote peer e7033baf2ef04118bf0ac6119f6d5ee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:39.489490 27209 raft_consensus.cc:1261] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09 [term 1 FOLLOWER]: Refusing update from remote peer e7033baf2ef04118bf0ac6119f6d5ee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:39.490456 27549 consensus_queue.cc:1035] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb8479a4e41448aaf53e2140f9f1f09" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 35551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:39.490801 27547 consensus_queue.cc:1035] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a249f39cc6ca43099575d570300ce650" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 35611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:39.518437 27063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:39.518774 27063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:39.523236 27063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b208d0b4db7741e2b23883754db2b949: Sending DeleteTablet for 3 replicas of tablet 9c1f388ddb974064ad78a180142ab593
I20260619 06:44:39.524477 27323 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c1f388ddb974064ad78a180142ab593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:39 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:57006
I20260619 06:44:39.524477 27189 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c1f388ddb974064ad78a180142ab593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:39 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:38776
I20260619 06:44:39.525453 27563 tablet_replica.cc:330] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: stopping tablet replica
I20260619 06:44:39.526015 27563 raft_consensus.cc:2229] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:39.526566 27563 raft_consensus.cc:2258] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:39.527115 27562 tablet_replica.cc:330] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: stopping tablet replica
I20260619 06:44:39.527141 27456 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c1f388ddb974064ad78a180142ab593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:39 UTC) from {username='kudu', principal='kudu/127.7.146.254@KRBTEST.COM'} at 127.0.0.1:47038
I20260619 06:44:39.527841 27562 raft_consensus.cc:2229] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:39.528138 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 27126
I20260619 06:44:39.528370 27562 raft_consensus.cc:2258] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:39.530565 27562 ts_tablet_manager.cc:1905] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:39.537653 27562 ts_tablet_manager.cc:1918] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:39.537909 27562 log.cc:1196] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/9c1f388ddb974064ad78a180142ab593
I20260619 06:44:39.538483 27562 ts_tablet_manager.cc:1939] T 9c1f388ddb974064ad78a180142ab593 P a249f39cc6ca43099575d570300ce650: Deleting consensus metadata
I20260619 06:44:39.540445 27049 catalog_manager.cc:4841] TS a249f39cc6ca43099575d570300ce650 (127.7.146.194:35611): tablet 9c1f388ddb974064ad78a180142ab593 (table test-table [id=5b046d9e5d4f41e988f00e3ad568a198]) successfully deleted
I20260619 06:44:39.541765 27563 ts_tablet_manager.cc:1905] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:39.542564 27564 tablet_replica.cc:330] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: stopping tablet replica
I20260619 06:44:39.543042 27564 raft_consensus.cc:2229] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:44:39.543740 27564 raft_consensus.cc:2258] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:39.545751 27564 ts_tablet_manager.cc:1905] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:39.549551 27563 ts_tablet_manager.cc:1918] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:39.549789 27563 log.cc:1196] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-0/wal/wals/9c1f388ddb974064ad78a180142ab593
Jun 19 06:44:39 dist-test-slave-qfk4 krb5kdc[27020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.193@KRBTEST.COM
I20260619 06:44:39.550397 27563 ts_tablet_manager.cc:1939] T 9c1f388ddb974064ad78a180142ab593 P abb8479a4e41448aaf53e2140f9f1f09: Deleting consensus metadata
I20260619 06:44:39.552433 27051 catalog_manager.cc:4841] TS abb8479a4e41448aaf53e2140f9f1f09 (127.7.146.193:35551): tablet 9c1f388ddb974064ad78a180142ab593 (table test-table [id=5b046d9e5d4f41e988f00e3ad568a198]) successfully deleted
I20260619 06:44:39.552989 27564 ts_tablet_manager.cc:1918] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:39.553167 27564 log.cc:1196] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/9c1f388ddb974064ad78a180142ab593
I20260619 06:44:39.553656 27564 ts_tablet_manager.cc:1939] T 9c1f388ddb974064ad78a180142ab593 P e7033baf2ef04118bf0ac6119f6d5ee2: Deleting consensus metadata
I20260619 06:44:39.555401 27051 catalog_manager.cc:4841] TS e7033baf2ef04118bf0ac6119f6d5ee2 (127.7.146.195:41859): tablet 9c1f388ddb974064ad78a180142ab593 (table test-table [id=5b046d9e5d4f41e988f00e3ad568a198]) successfully deleted
I20260619 06:44:39.558091 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.529655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.7.146.193:35551)
0619 06:44:39.529826 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:39.529832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:39.540028 (+ 10196us) server_negotiation.cc:407] Connection header received
0619 06:44:39.540325 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:39.540331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:39.540418 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:39.540508 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:39.542112 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.543261 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.545075 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.545662 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.545763 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.551362 (+ 5599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:39.551378 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:39.551396 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:39.551457 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:39.555153 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:39.555970 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:39.555981 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:39.555985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:39.556147 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:39.556545 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:39.556551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:39.556555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:39.557024 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:39.557336 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:39.557497 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 06:44:39.557720 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:44:39.558843 27249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:39.731007 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 27126
I20260619 06:44:39.745461 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 27260
Jun 19 06:44:39 dist-test-slave-qfk4 krb5kdc[27020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.194@KRBTEST.COM
I20260619 06:44:39.761713 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.746276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.7.146.194:35611)
0619 06:44:39.746479 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:39.746485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:39.746508 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:44:39.746646 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:39.746652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:39.746734 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:39.746857 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:39.748163 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.749251 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.750449 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.751061 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.751178 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.755621 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:39.755637 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:39.755653 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:39.755690 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:39.758639 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:39.759487 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:39.759495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:39.759497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:39.759632 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:39.760017 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:39.760021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:39.760023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:39.760428 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:39.760752 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:39.761089 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 06:44:39.761370 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:44:39.762274 27383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:39.879046 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 27260
I20260619 06:44:39.894353 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 27393
Jun 19 06:44:39 dist-test-slave-qfk4 krb5kdc[27020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.195@KRBTEST.COM
I20260619 06:44:39.910190 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:39.895102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.7.146.195:41859)
0619 06:44:39.895273 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:39.895280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:39.895400 (+ 120us) server_negotiation.cc:407] Connection header received
0619 06:44:39.895576 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:39.895580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:39.895646 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:39.895727 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:39.897195 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.898027 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.899282 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:39.899732 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:39.899848 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:39.904101 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:39.904115 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:39.904129 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:39.904177 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:39.907235 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:39.907992 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:39.907999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:39.908001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:39.908179 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:39.908602 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:39.908607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:39.908611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:39.908979 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:39.909308 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:39.909593 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 06:44:39.909820 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:44:39.910955 27516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:40.039422 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 27393
I20260619 06:44:40.056978 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 27033
I20260619 06:44:40.058084 27095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:40.169104 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 27033
2026-06-19T06:44:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4326 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T06:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:40Z Disabled control of system clock
I20260619 06:44:40.240911 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:43245
--webserver_interface=127.7.146.254
--webserver_port=0
--builtin_ntp_servers=127.7.146.212:39273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:43245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:44:40.567392 27573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:40.567725 27573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:40.567840 27573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:40.575318 27573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:40.575441 27573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:40.575502 27573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:40.575584 27573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:40.575634 27573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:40.585022 27573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:43245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:43245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:40.586308 27573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:40.588093 27573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:40.599660 27579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:40.599637 27578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:40.599792 27581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:40.600770 27573 server_base.cc:1034] running on GCE node
I20260619 06:44:40.601586 27573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:40.603063 27573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:40.604264 27573 hybrid_clock.cc:648] HybridClock initialized: now 1781851480604205 us; error 55 us; skew 500 ppm
I20260619 06:44:40.604650 27573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:40.606773 27573 webserver.cc:460] Webserver started at http://127.7.146.254:42955/ using document root <none> and password file <none>
I20260619 06:44:40.607445 27573 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:40.607543 27573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:40.607832 27573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:40.610646 27573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "4b181604f4fd417c983762836e072488"
format_stamp: "Formatted at 2026-06-19 06:44:40 on dist-test-slave-qfk4"
server_key: "98e6b26a5b19b899c63b56ac82ebbde6"
server_key_iv: "56f7249d2506ffb46555da33c12b9064"
server_key_version: "encryptionkey@0"
I20260619 06:44:40.611430 27573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "4b181604f4fd417c983762836e072488"
format_stamp: "Formatted at 2026-06-19 06:44:40 on dist-test-slave-qfk4"
server_key: "98e6b26a5b19b899c63b56ac82ebbde6"
server_key_iv: "56f7249d2506ffb46555da33c12b9064"
server_key_version: "encryptionkey@0"
I20260619 06:44:40.616321 27573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 06:44:40.619803 27587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:40.621243 27573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:44:40.621413 27573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "4b181604f4fd417c983762836e072488"
format_stamp: "Formatted at 2026-06-19 06:44:40 on dist-test-slave-qfk4"
server_key: "98e6b26a5b19b899c63b56ac82ebbde6"
server_key_iv: "56f7249d2506ffb46555da33c12b9064"
server_key_version: "encryptionkey@0"
I20260619 06:44:40.621616 27573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:40.633355 27573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:40.634135 27573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:40.634394 27573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:40.651625 27573 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:43245
I20260619 06:44:40.651648 27638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:43245 every 8 connection(s)
I20260619 06:44:40.653085 27573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:40.653712 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 27573
I20260619 06:44:40.653874 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:40.654423 7755 external_mini_cluster.cc:1428] Setting key b2cc9840713392b3ec117c86a8c197cc
I20260619 06:44:40.657454 27639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:40.669474 27639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488: Bootstrap starting.
I20260619 06:44:40.672012 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:40.656306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.7.146.254:43245)
0619 06:44:40.657319 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:40.657331 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:40.657374 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:44:40.659080 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:40.659103 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:40.659592 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:40.659956 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:40.660640 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:40.662046 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:40.663985 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:40.664768 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:40.664910 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:40.669512 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:40.669521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:40.669538 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:40.670176 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:40.670263 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:40.670288 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:40.670374 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:40.670630 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 06:44:40.671000 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":374,"threads_started":1}
I20260619 06:44:40.672945 27639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:40.674192 27639 log.cc:824] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:40.677251 27639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488: No bootstrap required, opened a new log
I20260619 06:44:40.685055 27639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b181604f4fd417c983762836e072488" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43245 } }
I20260619 06:44:40.685542 27639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:40.685626 27639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b181604f4fd417c983762836e072488, State: Initialized, Role: FOLLOWER
I20260619 06:44:40.686146 27639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b181604f4fd417c983762836e072488" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43245 } }
I20260619 06:44:40.686339 27639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:40.686437 27639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:40.686570 27639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:40.688719 27639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b181604f4fd417c983762836e072488" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43245 } }
I20260619 06:44:40.689229 27639 leader_election.cc:302] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b181604f4fd417c983762836e072488; no voters:
I20260619 06:44:40.689762 27639 leader_election.cc:288] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:40.689888 27644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:40.690619 27644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [term 1 LEADER]: Becoming Leader. State: Replica: 4b181604f4fd417c983762836e072488, State: Running, Role: LEADER
I20260619 06:44:40.691143 27644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b181604f4fd417c983762836e072488" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43245 } }
I20260619 06:44:40.691726 27639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:40.693674 27645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b181604f4fd417c983762836e072488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b181604f4fd417c983762836e072488" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43245 } } }
I20260619 06:44:40.694078 27645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:40.694501 27646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b181604f4fd417c983762836e072488. Latest consensus state: current_term: 1 leader_uuid: "4b181604f4fd417c983762836e072488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b181604f4fd417c983762836e072488" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 43245 } } }
I20260619 06:44:40.694860 27646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:40.697469 27649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:40.704054 27649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:40.713727 27649 catalog_manager.cc:1347] Generated new cluster ID: e99adfadc5ab4e8c909ddf1dd15dee17
I20260619 06:44:40.713845 27649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:40.723456 27649 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:40.724841 27649 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:40.736737 27649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488: Generated new TSK 0
I20260619 06:44:40.737646 27649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:40.748145 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.193:0
--local_ip_for_outbound_sockets=127.7.146.193
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43245
--builtin_ntp_servers=127.7.146.212:39273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:44:41.074066 27664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:41.074340 27664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:41.074468 27664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:41.081866 27664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:41.081980 27664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:41.082134 27664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.193
I20260619 06:44:41.091373 27664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.146.193
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.193
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:41.092905 27664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:41.094766 27664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:41.107807 27669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:41.107786 27670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:41.107743 27672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:41.108656 27664 server_base.cc:1034] running on GCE node
I20260619 06:44:41.109279 27664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:41.110247 27664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:41.111455 27664 hybrid_clock.cc:648] HybridClock initialized: now 1781851481111402 us; error 43 us; skew 500 ppm
I20260619 06:44:41.111761 27664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:41.114038 27664 webserver.cc:460] Webserver started at http://127.7.146.193:38213/ using document root <none> and password file <none>
I20260619 06:44:41.114722 27664 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:41.114810 27664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:41.115119 27664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:41.117990 27664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/data/instance:
uuid: "0ed4955bbc1745dc9a159993c2ff2764"
format_stamp: "Formatted at 2026-06-19 06:44:41 on dist-test-slave-qfk4"
server_key: "3f125dbe454e8400c74cf2cb7f0813a1"
server_key_iv: "b87003b9d556149078dd31b1b79049ff"
server_key_version: "encryptionkey@0"
I20260619 06:44:41.118846 27664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance:
uuid: "0ed4955bbc1745dc9a159993c2ff2764"
format_stamp: "Formatted at 2026-06-19 06:44:41 on dist-test-slave-qfk4"
server_key: "3f125dbe454e8400c74cf2cb7f0813a1"
server_key_iv: "b87003b9d556149078dd31b1b79049ff"
server_key_version: "encryptionkey@0"
I20260619 06:44:41.123843 27664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:44:41.127568 27678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:41.129122 27664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:44:41.129269 27664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/wal
uuid: "0ed4955bbc1745dc9a159993c2ff2764"
format_stamp: "Formatted at 2026-06-19 06:44:41 on dist-test-slave-qfk4"
server_key: "3f125dbe454e8400c74cf2cb7f0813a1"
server_key_iv: "b87003b9d556149078dd31b1b79049ff"
server_key_version: "encryptionkey@0"
I20260619 06:44:41.129468 27664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:41.158851 27664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:41.159698 27664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:41.159986 27664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:41.161006 27664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:41.162518 27664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:41.162606 27664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:41.162714 27664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:41.162784 27664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:41.192191 27664 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.193:42139
I20260619 06:44:41.192260 27790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.193:42139 every 8 connection(s)
I20260619 06:44:41.193589 27664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/data/info.pb
I20260619 06:44:41.201386 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 27664
I20260619 06:44:41.201541 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-0/wal/instance
I20260619 06:44:41.201941 7755 external_mini_cluster.cc:1428] Setting key 153877946f64ae2aed66d8e15522398b
I20260619 06:44:41.205036 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.194:0
--local_ip_for_outbound_sockets=127.7.146.194
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43245
--builtin_ntp_servers=127.7.146.212:39273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:44:41.207726 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:41.196142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.193:33633 (local address 127.7.146.254:43245)
0619 06:44:41.196616 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:41.196622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:41.198195 (+ 1573us) server_negotiation.cc:407] Connection header received
0619 06:44:41.200197 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:41.200202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:41.200278 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:41.200376 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:41.201726 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:41.202527 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:41.204561 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:41.205301 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:41.205428 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:41.206835 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:41.206842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:41.206847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:41.206873 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:41.206950 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:41.206967 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:41.207071 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:41.207272 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 06:44:41.207376 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":198,"threads_started":1}
I20260619 06:44:41.208809 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:41.196740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:43245 (local address 127.7.146.193:33633)
0619 06:44:41.198040 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:41.198078 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:41.199939 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:41.200710 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:41.200719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:41.200836 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:41.201512 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:41.201546 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:41.202721 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:41.202725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:41.204393 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:41.204406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:41.205520 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:41.205525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:41.205735 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:41.206479 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:44:41.206483 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:41.206542 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:44:41.206626 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:41.207066 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:41.207074 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:41.207081 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:44:41.207250 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 06:44:41.207631 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":294,"threads_started":1}
I20260619 06:44:41.213599 27791 heartbeater.cc:347] Connected to a master server at 127.7.146.254:43245
I20260619 06:44:41.213918 27791 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:41.214856 27791 heartbeater.cc:511] Master 127.7.146.254:43245 requested a full tablet report, sending...
I20260619 06:44:41.217273 27604 ts_manager.cc:194] Registered new tserver with Master: 0ed4955bbc1745dc9a159993c2ff2764 (127.7.146.193:42139)
I20260619 06:44:41.219146 27604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.193:33633
W20260619 06:44:41.525965 27795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:41.526242 27795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:41.526322 27795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:41.533434 27795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:41.533553 27795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:41.533692 27795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.194
I20260619 06:44:41.542830 27795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.146.194
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.194
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:41.544098 27795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:41.545831 27795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:41.558306 27800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:41.558338 27801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:41.558499 27803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:41.558991 27795 server_base.cc:1034] running on GCE node
I20260619 06:44:41.559564 27795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:41.560489 27795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:41.561728 27795 hybrid_clock.cc:648] HybridClock initialized: now 1781851481561671 us; error 49 us; skew 500 ppm
I20260619 06:44:41.562075 27795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:41.564280 27795 webserver.cc:460] Webserver started at http://127.7.146.194:44075/ using document root <none> and password file <none>
I20260619 06:44:41.565043 27795 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:41.565141 27795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:41.565435 27795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:41.568174 27795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/data/instance:
uuid: "37d3c84d66494955af58162a17f00e25"
format_stamp: "Formatted at 2026-06-19 06:44:41 on dist-test-slave-qfk4"
server_key: "30394f468abcb632cd466442ce474813"
server_key_iv: "8701a4d91b8ae844de934a932ed54661"
server_key_version: "encryptionkey@0"
I20260619 06:44:41.568943 27795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance:
uuid: "37d3c84d66494955af58162a17f00e25"
format_stamp: "Formatted at 2026-06-19 06:44:41 on dist-test-slave-qfk4"
server_key: "30394f468abcb632cd466442ce474813"
server_key_iv: "8701a4d91b8ae844de934a932ed54661"
server_key_version: "encryptionkey@0"
I20260619 06:44:41.573873 27795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 06:44:41.577364 27809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:41.578773 27795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:44:41.578930 27795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/wal
uuid: "37d3c84d66494955af58162a17f00e25"
format_stamp: "Formatted at 2026-06-19 06:44:41 on dist-test-slave-qfk4"
server_key: "30394f468abcb632cd466442ce474813"
server_key_iv: "8701a4d91b8ae844de934a932ed54661"
server_key_version: "encryptionkey@0"
I20260619 06:44:41.579144 27795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:41.602054 27795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:41.602867 27795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:41.603147 27795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:41.604085 27795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:41.605569 27795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:41.605741 27795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:41.605904 27795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:41.605989 27795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:41.635993 27795 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.194:43879
I20260619 06:44:41.636057 27921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.194:43879 every 8 connection(s)
I20260619 06:44:41.637255 27795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/data/info.pb
I20260619 06:44:41.641754 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 27795
I20260619 06:44:41.641922 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/wal/instance
I20260619 06:44:41.642483 7755 external_mini_cluster.cc:1428] Setting key 1a13656ca0969c18e76c4e68e46d6239
I20260619 06:44:41.645556 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.146.195:0
--local_ip_for_outbound_sockets=127.7.146.195
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43245
--builtin_ntp_servers=127.7.146.212:39273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:44:41.650857 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:41.639885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:60995 (local address 127.7.146.254:43245)
0619 06:44:41.640051 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:41.640055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:41.641595 (+ 1540us) server_negotiation.cc:407] Connection header received
0619 06:44:41.643844 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:41.643848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:41.643912 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:41.644001 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:41.645332 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:41.646106 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:41.648176 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:41.648602 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:41.648692 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:41.650148 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:41.650152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:41.650156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:41.650174 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:41.650206 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:41.650217 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:41.650276 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:41.650497 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 06:44:41.650579 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:44:41.652148 27924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:41.640434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:43245 (local address 127.7.146.194:60995)
0619 06:44:41.641381 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:41.641429 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:41.643560 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:41.644261 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:41.644274 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:41.644382 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:41.645170 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:41.645195 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:41.646260 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:41.646266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:41.648058 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:41.648072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:41.648810 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:41.648816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:41.649075 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:41.649791 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:44:41.649795 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:41.649843 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:44:41.649951 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:41.650290 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:41.650299 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:41.650306 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:44:41.650486 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 06:44:41.650887 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":276,"threads_started":1}
I20260619 06:44:41.653745 27922 heartbeater.cc:347] Connected to a master server at 127.7.146.254:43245
I20260619 06:44:41.654076 27922 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:41.654788 27922 heartbeater.cc:511] Master 127.7.146.254:43245 requested a full tablet report, sending...
I20260619 06:44:41.656181 27604 ts_manager.cc:194] Registered new tserver with Master: 37d3c84d66494955af58162a17f00e25 (127.7.146.194:43879)
I20260619 06:44:41.657322 27604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.194:60995
W20260619 06:44:41.969861 27925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:41.970168 27925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:41.970297 27925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:41.977770 27925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:41.977898 27925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:41.978044 27925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.146.195
I20260619 06:44:41.987376 27925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:39273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.146.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.146.195
--webserver_port=0
--tserver_master_addrs=127.7.146.254:43245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.146.195
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:41.988669 27925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:41.990445 27925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:42.003347 27931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:42.003378 27933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:42.003398 27930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:42.004048 27925 server_base.cc:1034] running on GCE node
I20260619 06:44:42.004746 27925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:42.005721 27925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:42.006950 27925 hybrid_clock.cc:648] HybridClock initialized: now 1781851482006894 us; error 51 us; skew 500 ppm
I20260619 06:44:42.007277 27925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:44:42.009821 27925 webserver.cc:460] Webserver started at http://127.7.146.195:32997/ using document root <none> and password file <none>
I20260619 06:44:42.010536 27925 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:42.010639 27925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:42.010974 27925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:42.013953 27925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/data/instance:
uuid: "ce0e6df863354e07946881611c95d378"
format_stamp: "Formatted at 2026-06-19 06:44:42 on dist-test-slave-qfk4"
server_key: "3f0f262ececb1e0b353836ce5fa32b96"
server_key_iv: "95a88473468070685e0255e4cf1b8edf"
server_key_version: "encryptionkey@0"
I20260619 06:44:42.014748 27925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance:
uuid: "ce0e6df863354e07946881611c95d378"
format_stamp: "Formatted at 2026-06-19 06:44:42 on dist-test-slave-qfk4"
server_key: "3f0f262ececb1e0b353836ce5fa32b96"
server_key_iv: "95a88473468070685e0255e4cf1b8edf"
server_key_version: "encryptionkey@0"
I20260619 06:44:42.019927 27925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 06:44:42.023556 27939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:42.024982 27925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:44:42.025133 27925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/wal
uuid: "ce0e6df863354e07946881611c95d378"
format_stamp: "Formatted at 2026-06-19 06:44:42 on dist-test-slave-qfk4"
server_key: "3f0f262ececb1e0b353836ce5fa32b96"
server_key_iv: "95a88473468070685e0255e4cf1b8edf"
server_key_version: "encryptionkey@0"
I20260619 06:44:42.025327 27925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:42.039378 27925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:42.040198 27925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:42.040482 27925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:42.041465 27925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:44:42.042994 27925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:44:42.043092 27925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:42.043195 27925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:44:42.043262 27925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:42.074024 27925 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.195:45019
I20260619 06:44:42.074095 28051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.195:45019 every 8 connection(s)
I20260619 06:44:42.075340 27925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/data/info.pb
I20260619 06:44:42.083514 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 27925
I20260619 06:44:42.083694 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/wal/instance
I20260619 06:44:42.084213 7755 external_mini_cluster.cc:1428] Setting key 15250c04e4e134211f121ce4758901bc
I20260619 06:44:42.088511 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.077984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.195:55215 (local address 127.7.146.254:43245)
0619 06:44:42.078191 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:42.078198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:42.079630 (+ 1432us) server_negotiation.cc:407] Connection header received
0619 06:44:42.081560 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:42.081565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:42.081642 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:42.081749 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:42.083005 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.084088 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.086007 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.086424 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.086523 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.087904 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:42.087908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:42.087912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:42.087930 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:42.087968 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:42.087980 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:42.088039 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:42.088199 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 06:44:42.088278 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:44:42.089622 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.078564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:43245 (local address 127.7.146.195:55215)
0619 06:44:42.079470 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.079514 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:42.081212 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:42.081972 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:42.081981 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:42.082079 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:44:42.082779 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.082798 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.084251 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.084255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.085865 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.085877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.086651 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.086659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.086959 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.087598 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:44:42.087601 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:44:42.087642 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:44:42.087726 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:44:42.088045 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:44:42.088051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:44:42.088057 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:44:42.088190 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 06:44:42.088520 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":276,"threads_started":1}
I20260619 06:44:42.091006 28052 heartbeater.cc:347] Connected to a master server at 127.7.146.254:43245
I20260619 06:44:42.091308 28052 heartbeater.cc:464] Registering TS with master...
I20260619 06:44:42.092140 28052 heartbeater.cc:511] Master 127.7.146.254:43245 requested a full tablet report, sending...
I20260619 06:44:42.093600 27604 ts_manager.cc:194] Registered new tserver with Master: ce0e6df863354e07946881611c95d378 (127.7.146.195:45019)
I20260619 06:44:42.094463 27604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.146.195:55215
I20260619 06:44:42.099205 7755 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:44:42.108542 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.103197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.7.146.254:43245)
0619 06:44:42.103355 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:42.103361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:42.103745 (+ 384us) server_negotiation.cc:407] Connection header received
0619 06:44:42.103901 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:42.103905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:42.103981 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:42.104059 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:42.104521 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.105268 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.106577 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.106992 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.107087 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.107975 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:42.107979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:42.108006 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:42.108025 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:42.108057 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:42.108069 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:42.108133 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:42.108215 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 06:44:42.108306 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:44:42.112087 27604 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:44:42.119107 27604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:44:42.143024 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.132194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:45019 (local address 127.0.0.1:39974)
0619 06:44:42.133028 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.133071 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:42.133225 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:42.134666 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:42.134672 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:42.134700 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:42.135052 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.135063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.137089 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.137095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.139457 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.139474 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.142080 (+ 2606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.142087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.142282 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.142346 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 06:44:42.142489 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 06:44:42.142681 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":401,"threads_started":1}
I20260619 06:44:42.143102 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.133359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.7.146.195:45019)
0619 06:44:42.134046 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:42.134055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:42.134088 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:44:42.134187 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:42.134195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:42.134463 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:42.134661 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:42.135219 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.136938 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.140693 (+ 3755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.141857 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.142036 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.142322 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:42.142553 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 06:44:42.142727 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":921,"threads_started":1}
I20260619 06:44:42.143976 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.132557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:42139 (local address 127.0.0.1:38128)
0619 06:44:42.133930 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.133953 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:42.136652 (+ 2699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:42.137391 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:42.137396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:42.137421 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:42.137913 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.137927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.139576 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.139581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.141134 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.141149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.143204 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.143210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.143414 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.143443 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:44:42.143534 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:44:42.143663 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1246,"mutex_wait_us":44,"thread_start_us":1199,"threads_started":1}
I20260619 06:44:42.144717 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.132985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.7.146.193:42139)
0619 06:44:42.133450 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:42.133459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:42.134109 (+ 650us) server_negotiation.cc:407] Connection header received
0619 06:44:42.136934 (+ 2825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:42.136943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:42.137201 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:42.137392 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:42.138147 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.139458 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.141335 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.142723 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.142857 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.143077 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:42.144248 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0619 06:44:42.144386 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":621,"threads_started":1}
I20260619 06:44:42.145243 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.134148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.194:43879 (local address 127.0.0.1:44322)
0619 06:44:42.134624 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.134641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:42.134764 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:42.137711 (+ 2947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:42.137715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:42.137742 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:42.138259 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.138271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.140203 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.140208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.141786 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.141800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.144517 (+ 2717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.144523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.144725 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.144756 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:44:42.144850 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:44:42.144970 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":269,"threads_started":1}
I20260619 06:44:42.145386 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.134469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.7.146.194:43879)
0619 06:44:42.137139 (+ 2670us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:42.137155 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:42.137184 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:44:42.137272 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:42.137279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:42.137518 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:42.137710 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:42.138490 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.140072 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.141979 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.144312 (+ 2333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.144457 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.144777 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:42.144891 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 06:44:42.145051 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2512,"thread_start_us":2388,"threads_started":1}
I20260619 06:44:42.147439 27987 tablet_service.cc:1467] Processing CreateTablet for tablet b31de5619b914c22b66b08fe5d51304e (DEFAULT_TABLE table=test-table [id=e49f8a34607242d8993597bcf5d2b664]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:42.148252 27726 tablet_service.cc:1467] Processing CreateTablet for tablet b31de5619b914c22b66b08fe5d51304e (DEFAULT_TABLE table=test-table [id=e49f8a34607242d8993597bcf5d2b664]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:42.148967 27987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31de5619b914c22b66b08fe5d51304e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:42.149075 27857 tablet_service.cc:1467] Processing CreateTablet for tablet b31de5619b914c22b66b08fe5d51304e (DEFAULT_TABLE table=test-table [id=e49f8a34607242d8993597bcf5d2b664]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:44:42.149685 27726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31de5619b914c22b66b08fe5d51304e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:42.150373 27857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31de5619b914c22b66b08fe5d51304e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:42.163252 28070 tablet_bootstrap.cc:492] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764: Bootstrap starting.
I20260619 06:44:42.167268 28070 tablet_bootstrap.cc:654] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:42.167778 28071 tablet_bootstrap.cc:492] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378: Bootstrap starting.
I20260619 06:44:42.168816 28070 log.cc:824] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:42.171452 28072 tablet_bootstrap.cc:492] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25: Bootstrap starting.
I20260619 06:44:42.173065 28071 tablet_bootstrap.cc:654] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:42.174460 28071 log.cc:824] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:42.175374 28072 tablet_bootstrap.cc:654] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:42.175974 28070 tablet_bootstrap.cc:492] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764: No bootstrap required, opened a new log
I20260619 06:44:42.176306 28070 ts_tablet_manager.cc:1397] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260619 06:44:42.176834 28072 log.cc:824] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:42.178071 28071 tablet_bootstrap.cc:492] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378: No bootstrap required, opened a new log
I20260619 06:44:42.178355 28071 ts_tablet_manager.cc:1397] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 06:44:42.180359 28072 tablet_bootstrap.cc:492] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25: No bootstrap required, opened a new log
I20260619 06:44:42.180702 28072 ts_tablet_manager.cc:1397] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 06:44:42.186750 28071 raft_consensus.cc:348] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0e6df863354e07946881611c95d378" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45019 } } peers { permanent_uuid: "0ed4955bbc1745dc9a159993c2ff2764" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42139 } } peers { permanent_uuid: "37d3c84d66494955af58162a17f00e25" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 43879 } }
I20260619 06:44:42.187235 28071 raft_consensus.cc:374] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:42.187361 28071 raft_consensus.cc:729] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0e6df863354e07946881611c95d378, State: Initialized, Role: FOLLOWER
I20260619 06:44:42.188005 28071 consensus_queue.cc:260] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0e6df863354e07946881611c95d378" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45019 } } peers { permanent_uuid: "0ed4955bbc1745dc9a159993c2ff2764" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42139 } } peers { permanent_uuid: "37d3c84d66494955af58162a17f00e25" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 43879 } }
I20260619 06:44:42.188160 28070 raft_consensus.cc:348] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0e6df863354e07946881611c95d378" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45019 } } peers { permanent_uuid: "0ed4955bbc1745dc9a159993c2ff2764" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42139 } } peers { permanent_uuid: "37d3c84d66494955af58162a17f00e25" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 43879 } }
I20260619 06:44:42.188658 28070 raft_consensus.cc:374] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:42.188786 28070 raft_consensus.cc:729] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed4955bbc1745dc9a159993c2ff2764, State: Initialized, Role: FOLLOWER
I20260619 06:44:42.189266 28052 heartbeater.cc:503] Master 127.7.146.254:43245 was elected leader, sending a full tablet report...
I20260619 06:44:42.189517 28070 consensus_queue.cc:260] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0e6df863354e07946881611c95d378" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45019 } } peers { permanent_uuid: "0ed4955bbc1745dc9a159993c2ff2764" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42139 } } peers { permanent_uuid: "37d3c84d66494955af58162a17f00e25" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 43879 } }
I20260619 06:44:42.189917 28071 ts_tablet_manager.cc:1428] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260619 06:44:42.190688 27791 heartbeater.cc:503] Master 127.7.146.254:43245 was elected leader, sending a full tablet report...
I20260619 06:44:42.190486 28072 raft_consensus.cc:348] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0e6df863354e07946881611c95d378" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45019 } } peers { permanent_uuid: "0ed4955bbc1745dc9a159993c2ff2764" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42139 } } peers { permanent_uuid: "37d3c84d66494955af58162a17f00e25" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 43879 } }
I20260619 06:44:42.190968 28072 raft_consensus.cc:374] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:42.191063 28072 raft_consensus.cc:729] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d3c84d66494955af58162a17f00e25, State: Initialized, Role: FOLLOWER
I20260619 06:44:42.191185 28070 ts_tablet_manager.cc:1428] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260619 06:44:42.192603 28072 consensus_queue.cc:260] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0e6df863354e07946881611c95d378" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45019 } } peers { permanent_uuid: "0ed4955bbc1745dc9a159993c2ff2764" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42139 } } peers { permanent_uuid: "37d3c84d66494955af58162a17f00e25" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 43879 } }
I20260619 06:44:42.194078 27922 heartbeater.cc:503] Master 127.7.146.254:43245 was elected leader, sending a full tablet report...
I20260619 06:44:42.194684 28072 ts_tablet_manager.cc:1428] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25: Time spent starting tablet: real 0.014s user 0.007s sys 0.003s
W20260619 06:44:42.195843 27792 tablet.cc:2369] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:44:42.225023 28078 raft_consensus.cc:482] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:44:42.225317 28078 raft_consensus.cc:504] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0e6df863354e07946881611c95d378" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45019 } } peers { permanent_uuid: "0ed4955bbc1745dc9a159993c2ff2764" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42139 } } peers { permanent_uuid: "37d3c84d66494955af58162a17f00e25" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 43879 } }
I20260619 06:44:42.227005 28078 leader_election.cc:288] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce0e6df863354e07946881611c95d378 (127.7.146.195:45019), 0ed4955bbc1745dc9a159993c2ff2764 (127.7.146.193:42139)
I20260619 06:44:42.235041 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.227509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:40847 (local address 127.7.146.193:42139)
0619 06:44:42.227723 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:42.227729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:42.228447 (+ 718us) server_negotiation.cc:407] Connection header received
0619 06:44:42.228557 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:42.228584 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:42.228674 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:42.228751 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:42.229419 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.230523 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.233367 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.234207 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.234346 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.234410 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:42.234663 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 06:44:42.234787 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:44:42.235064 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.227431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:42139 (local address 127.7.146.194:40847)
0619 06:44:42.228329 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.228352 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:42.228460 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:42.228791 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:42.228795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:42.228830 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:42.229268 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.229279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.230648 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.230653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.233169 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.233184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.234356 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.234360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.234508 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.234532 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:44:42.234610 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 06:44:42.234732 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":105,"thread_start_us":579,"threads_started":1}
I20260619 06:44:42.236159 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.227494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.146.194:42149 (local address 127.7.146.195:45019)
0619 06:44:42.227887 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:42.227892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:42.228225 (+ 333us) server_negotiation.cc:407] Connection header received
0619 06:44:42.228449 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:42.228454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:42.228544 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:42.228662 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:44:42.229337 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.230523 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.234500 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.235335 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.235487 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.235557 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:42.235814 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 06:44:42.235930 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260619 06:44:42.236171 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.227358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.195:45019 (local address 127.7.146.194:42149)
0619 06:44:42.228085 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.228108 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:44:42.228299 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:44:42.228695 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:44:42.228701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:44:42.228728 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:44:42.229199 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.229212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.230647 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.230653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.233169 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:44:42.233184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.235513 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.235517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:44:42.235669 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.235699 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:44:42.235787 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:44:42.235906 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":168,"spinlock_wait_cycles":2944,"thread_start_us":351,"threads_started":1}
I20260619 06:44:42.236418 27746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31de5619b914c22b66b08fe5d51304e" candidate_uuid: "37d3c84d66494955af58162a17f00e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed4955bbc1745dc9a159993c2ff2764" is_pre_election: true
I20260619 06:44:42.237056 27746 raft_consensus.cc:2454] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37d3c84d66494955af58162a17f00e25 in term 0.
I20260619 06:44:42.237217 28007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31de5619b914c22b66b08fe5d51304e" candidate_uuid: "37d3c84d66494955af58162a17f00e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0e6df863354e07946881611c95d378" is_pre_election: true
I20260619 06:44:42.237748 28007 raft_consensus.cc:2454] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37d3c84d66494955af58162a17f00e25 in term 0.
I20260619 06:44:42.237762 27813 leader_election.cc:302] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed4955bbc1745dc9a159993c2ff2764, 37d3c84d66494955af58162a17f00e25; no voters:
I20260619 06:44:42.238158 28078 raft_consensus.cc:2781] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:44:42.238310 28078 raft_consensus.cc:482] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:44:42.238408 28078 raft_consensus.cc:3037] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:42.240867 28078 raft_consensus.cc:504] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0e6df863354e07946881611c95d378" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45019 } } peers { permanent_uuid: "0ed4955bbc1745dc9a159993c2ff2764" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42139 } } peers { permanent_uuid: "37d3c84d66494955af58162a17f00e25" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 43879 } }
I20260619 06:44:42.241392 28078 leader_election.cc:288] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [CANDIDATE]: Term 1 election: Requested vote from peers ce0e6df863354e07946881611c95d378 (127.7.146.195:45019), 0ed4955bbc1745dc9a159993c2ff2764 (127.7.146.193:42139)
I20260619 06:44:42.241868 27746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31de5619b914c22b66b08fe5d51304e" candidate_uuid: "37d3c84d66494955af58162a17f00e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed4955bbc1745dc9a159993c2ff2764"
I20260619 06:44:42.241868 28007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31de5619b914c22b66b08fe5d51304e" candidate_uuid: "37d3c84d66494955af58162a17f00e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0e6df863354e07946881611c95d378"
I20260619 06:44:42.242067 28007 raft_consensus.cc:3037] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:42.242072 27746 raft_consensus.cc:3037] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:42.245154 27746 raft_consensus.cc:2454] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37d3c84d66494955af58162a17f00e25 in term 1.
I20260619 06:44:42.245154 28007 raft_consensus.cc:2454] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37d3c84d66494955af58162a17f00e25 in term 1.
I20260619 06:44:42.245561 27812 leader_election.cc:302] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ed4955bbc1745dc9a159993c2ff2764, 37d3c84d66494955af58162a17f00e25, ce0e6df863354e07946881611c95d378; no voters:
I20260619 06:44:42.245870 28078 raft_consensus.cc:2781] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:42.246266 28078 raft_consensus.cc:686] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 1 LEADER]: Becoming Leader. State: Replica: 37d3c84d66494955af58162a17f00e25, State: Running, Role: LEADER
I20260619 06:44:42.246737 28078 consensus_queue.cc:237] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0e6df863354e07946881611c95d378" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45019 } } peers { permanent_uuid: "0ed4955bbc1745dc9a159993c2ff2764" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42139 } } peers { permanent_uuid: "37d3c84d66494955af58162a17f00e25" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 43879 } }
I20260619 06:44:42.251128 27603 catalog_manager.cc:5495] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37d3c84d66494955af58162a17f00e25 (127.7.146.194). New cstate: current_term: 1 leader_uuid: "37d3c84d66494955af58162a17f00e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0e6df863354e07946881611c95d378" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ed4955bbc1745dc9a159993c2ff2764" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37d3c84d66494955af58162a17f00e25" member_type: VOTER last_known_addr { host: "127.7.146.194" port: 43879 } health_report { overall_health: HEALTHY } } }
I20260619 06:44:42.268527 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.263402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.7.146.194:43879)
0619 06:44:42.263585 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:42.263590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:42.263607 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:44:42.263762 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:42.263765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:42.263839 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:42.263959 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:44:42.264242 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.264992 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.266500 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.266893 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.266983 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.267505 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:44:42.267988 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:44:42.268090 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:42.268203 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 06:44:42.268270 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:44:42.276824 28007 raft_consensus.cc:1261] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378 [term 1 FOLLOWER]: Refusing update from remote peer 37d3c84d66494955af58162a17f00e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:42.276824 27746 raft_consensus.cc:1261] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764 [term 1 FOLLOWER]: Refusing update from remote peer 37d3c84d66494955af58162a17f00e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:44:42.277988 28081 consensus_queue.cc:1035] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed4955bbc1745dc9a159993c2ff2764" member_type: VOTER last_known_addr { host: "127.7.146.193" port: 42139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:42.278414 28078 consensus_queue.cc:1035] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce0e6df863354e07946881611c95d378" member_type: VOTER last_known_addr { host: "127.7.146.195" port: 45019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:44:42.289690 28084 mvcc.cc:205] Tried to move back new op lower bound from 7298463671395758080 to 7298463671286374400. Current Snapshot: MvccSnapshot[applied={T|T < 7298463671395758080}]
I20260619 06:44:42.305346 27604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:42.305639 27604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:44:42.308980 27604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b181604f4fd417c983762836e072488: Sending DeleteTablet for 3 replicas of tablet b31de5619b914c22b66b08fe5d51304e
I20260619 06:44:42.310180 27857 tablet_service.cc:1514] Processing DeleteTablet for tablet b31de5619b914c22b66b08fe5d51304e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:42 UTC) from {username='slave'} at 127.0.0.1:44322
I20260619 06:44:42.310151 27987 tablet_service.cc:1514] Processing DeleteTablet for tablet b31de5619b914c22b66b08fe5d51304e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:42 UTC) from {username='slave'} at 127.0.0.1:39974
I20260619 06:44:42.311195 27726 tablet_service.cc:1514] Processing DeleteTablet for tablet b31de5619b914c22b66b08fe5d51304e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:44:42 UTC) from {username='slave'} at 127.0.0.1:38128
I20260619 06:44:42.311887 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 27664
I20260619 06:44:42.321532 28094 tablet_replica.cc:330] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25: stopping tablet replica
I20260619 06:44:42.322093 28094 raft_consensus.cc:2229] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:44:42.322852 28094 raft_consensus.cc:2258] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:42.325556 28094 ts_tablet_manager.cc:1905] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:42.326416 28095 tablet_replica.cc:330] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378: stopping tablet replica
I20260619 06:44:42.326777 28095 raft_consensus.cc:2229] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:42.327252 28095 raft_consensus.cc:2258] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:42.330574 28095 ts_tablet_manager.cc:1905] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:42.334483 28094 ts_tablet_manager.cc:1918] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:42.334718 28094 log.cc:1196] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-1/wal/wals/b31de5619b914c22b66b08fe5d51304e
I20260619 06:44:42.335338 28094 ts_tablet_manager.cc:1939] T b31de5619b914c22b66b08fe5d51304e P 37d3c84d66494955af58162a17f00e25: Deleting consensus metadata
I20260619 06:44:42.337347 27591 catalog_manager.cc:4841] TS 37d3c84d66494955af58162a17f00e25 (127.7.146.194:43879): tablet b31de5619b914c22b66b08fe5d51304e (table test-table [id=e49f8a34607242d8993597bcf5d2b664]) successfully deleted
I20260619 06:44:42.337565 28095 ts_tablet_manager.cc:1918] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:44:42.337711 28095 log.cc:1196] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378: Deleting WAL directory at /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851174422545-7755-0/minicluster-data/ts-2/wal/wals/b31de5619b914c22b66b08fe5d51304e
I20260619 06:44:42.338160 28095 ts_tablet_manager.cc:1939] T b31de5619b914c22b66b08fe5d51304e P ce0e6df863354e07946881611c95d378: Deleting consensus metadata
I20260619 06:44:42.339208 28096 tablet_replica.cc:330] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764: stopping tablet replica
I20260619 06:44:42.339735 28096 raft_consensus.cc:2229] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:44:42.339828 27590 catalog_manager.cc:4841] TS ce0e6df863354e07946881611c95d378 (127.7.146.195:45019): tablet b31de5619b914c22b66b08fe5d51304e (table test-table [id=e49f8a34607242d8993597bcf5d2b664]) successfully deleted
I20260619 06:44:42.340333 28096 raft_consensus.cc:2258] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:44:42.342346 28096 ts_tablet_manager.cc:1905] T b31de5619b914c22b66b08fe5d51304e P 0ed4955bbc1745dc9a159993c2ff2764: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:44:42.344637 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.313254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.7.146.193:42139)
0619 06:44:42.313450 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:42.313456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:42.337257 (+ 23801us) server_negotiation.cc:407] Connection header received
0619 06:44:42.337781 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:42.337787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:42.337925 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:42.338011 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:42.338736 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.339886 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.341623 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.342044 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.342152 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.343169 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:42.343177 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:42.343190 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:42.343779 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:42.343829 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:42.343850 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:42.344054 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:42.344167 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 06:44:42.344320 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:44:42.345359 27786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:42.435797 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 27664
W20260619 06:44:42.457360 27591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.146.193:42139 (error 108)
I20260619 06:44:42.458010 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 27795
I20260619 06:44:42.459344 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.458802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:42139 (local address 127.0.0.1:38144)
0619 06:44:42.459046 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.459174 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:42139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260619 06:44:42.459604 27591 catalog_manager.cc:4584] TS 0ed4955bbc1745dc9a159993c2ff2764 (127.7.146.193:42139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b31de5619b914c22b66b08fe5d51304e: Network error: Client connection negotiation failed: client connection to 127.7.146.193:42139: connect: Connection refused (error 111)
I20260619 06:44:42.465236 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.458815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.7.146.194:43879)
0619 06:44:42.459029 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:42.459036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:42.459129 (+ 93us) server_negotiation.cc:407] Connection header received
0619 06:44:42.459315 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:42.459321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:42.459428 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:42.459541 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:42.460073 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.460903 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.462111 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.462558 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.462664 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.463786 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:42.463795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:42.463810 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:42.464419 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:42.464472 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:42.464491 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:42.464612 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:42.464810 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 06:44:42.464917 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:44:42.465937 27917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:42.513236 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.512777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:42139 (local address 127.0.0.1:38160)
0619 06:44:42.512969 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.513070 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:42139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:44:42.514279 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.513927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:42139 (local address 127.0.0.1:38168)
0619 06:44:42.514079 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.514164 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:42139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 06:44:42.575012 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.574497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:42139 (local address 127.0.0.1:38180)
0619 06:44:42.574724 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.574824 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:42139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 06:44:42.576085 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.575686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:42139 (local address 127.0.0.1:38184)
0619 06:44:42.575859 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.575959 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:42139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 06:44:42.578490 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 27795
I20260619 06:44:42.593473 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 27925
I20260619 06:44:42.600087 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.594248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.7.146.195:45019)
0619 06:44:42.594435 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:42.594441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:42.594460 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:44:42.594632 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:42.594637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:42.594726 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:42.594809 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:42.595203 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.595913 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.597220 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:42.597612 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:42.597719 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:42.598679 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:42.598689 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:42.598703 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:42.599377 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:42.599427 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:42.599449 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:42.599561 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:42.599682 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 06:44:42.599791 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:44:42.600726 28047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:42.657039 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.656475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:42139 (local address 127.0.0.1:38188)
0619 06:44:42.656731 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.656855 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:42139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 06:44:42.658229 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:42.657771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.193:42139 (local address 127.0.0.1:38192)
0619 06:44:42.657993 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 06:44:42.658084 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.193:42139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 06:44:42.689788 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 27925
I20260619 06:44:42.702731 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 27573
I20260619 06:44:42.703797 27634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:42.812804 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 27573
2026-06-19T06:44:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2641 ms)
[----------] 6 tests from EncryptionPolicyTest (20771 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:42 dist-test-slave-qfk4 krb5kdc[28105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:42 dist-test-slave-qfk4 krb5kdc[28105](info): set up 2 sockets
Jun 19 06:44:42 dist-test-slave-qfk4 krb5kdc[28105](info): commencing operation
krb5kdc: starting...
W20260619 06:44:43.832928 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:43 dist-test-slave-qfk4 krb5kdc[28105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:44:44.075721 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:44.397971 28118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:44.398289 28118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:44.398434 28118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:44.405879 28118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:44.406023 28118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:44.406102 28118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:44.406188 28118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:44.406244 28118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:44.416323 28118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:44.417651 28118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:44.418965 28118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:44.430680 28126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:44.430711 28124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:44.430739 28123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:44.431514 28118 server_base.cc:1034] running on GCE node
I20260619 06:44:44.432307 28118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:44.433712 28118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:44.434937 28118 hybrid_clock.cc:648] HybridClock initialized: now 1781851484434884 us; error 43 us; skew 500 ppm
I20260619 06:44:44.435267 28118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:44 dist-test-slave-qfk4 krb5kdc[28105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:44.438314 28118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:44:44.439677 28118 webserver.cc:460] Webserver started at http://127.0.0.1:41867/ using document root <none> and password file <none>
I20260619 06:44:44.440343 28118 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:44.440456 28118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:44.440783 28118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:44.443694 28118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "147f00906ff143c7af29ec48dff92a78"
format_stamp: "Formatted at 2026-06-19 06:44:44 on dist-test-slave-qfk4"
server_key: "7fe3a6a11d1d926568bc0dd0374cc65b"
server_key_iv: "47bb8870922cdff0e49c531b0c72b335"
server_key_version: "encryptionkey@0"
I20260619 06:44:44.444464 28118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "147f00906ff143c7af29ec48dff92a78"
format_stamp: "Formatted at 2026-06-19 06:44:44 on dist-test-slave-qfk4"
server_key: "7fe3a6a11d1d926568bc0dd0374cc65b"
server_key_iv: "47bb8870922cdff0e49c531b0c72b335"
server_key_version: "encryptionkey@0"
I20260619 06:44:44.449605 28118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:44:44.453187 28133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:44.454581 28118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:44:44.454769 28118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "147f00906ff143c7af29ec48dff92a78"
format_stamp: "Formatted at 2026-06-19 06:44:44 on dist-test-slave-qfk4"
server_key: "7fe3a6a11d1d926568bc0dd0374cc65b"
server_key_iv: "47bb8870922cdff0e49c531b0c72b335"
server_key_version: "encryptionkey@0"
I20260619 06:44:44.454989 28118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:44.477873 28118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:44.482673 28118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:44.482973 28118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:44.501681 28118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39225
I20260619 06:44:44.501705 28184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39225 every 8 connection(s)
I20260619 06:44:44.503135 28118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:44.506912 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:44.508368 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 28118
I20260619 06:44:44.508615 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:44.509248 7755 external_mini_cluster.cc:1428] Setting key 55c98c8b3737b84f429627fa1d66ec71
I20260619 06:44:44.520936 28185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78: Bootstrap starting.
Jun 19 06:44:44 dist-test-slave-qfk4 krb5kdc[28105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:44:44.524082 28185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:44.525260 28185 log.cc:824] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:44.528549 28185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78: No bootstrap required, opened a new log
I20260619 06:44:44.533012 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:44.511223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38177 (local address 127.0.0.1:39225)
0619 06:44:44.512034 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:44.512049 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:44.512110 (+ 61us) server_negotiation.cc:407] Connection header received
0619 06:44:44.513905 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:44.513939 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:44.514490 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:44.514911 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:44.516207 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:44.518265 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:44.519615 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:44.520644 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:44.520813 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:44.524872 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:44.524882 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:44.524902 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:44.524951 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:44.528429 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:44.529353 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:44.529361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:44.529369 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:44.529566 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:44.529945 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:44.529951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:44.529954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:44.530633 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:44.530945 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:44.531286 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 06:44:44.531624 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":340,"threads_started":1}
I20260619 06:44:44.537319 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:44:44.538189 28185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147f00906ff143c7af29ec48dff92a78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260619 06:44:44.538535 28185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:44.538631 28185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147f00906ff143c7af29ec48dff92a78, State: Initialized, Role: FOLLOWER
I20260619 06:44:44.539211 28185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147f00906ff143c7af29ec48dff92a78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260619 06:44:44.539415 28185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:44.539536 28185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:44.539690 28185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:44.542315 28185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147f00906ff143c7af29ec48dff92a78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260619 06:44:44.542850 28185 leader_election.cc:302] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 147f00906ff143c7af29ec48dff92a78; no voters:
I20260619 06:44:44.543431 28185 leader_election.cc:288] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:44.543534 28195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:44.544795 28195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [term 1 LEADER]: Becoming Leader. State: Replica: 147f00906ff143c7af29ec48dff92a78, State: Running, Role: LEADER
I20260619 06:44:44.545442 28195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147f00906ff143c7af29ec48dff92a78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } }
I20260619 06:44:44.545802 28185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:44.548169 28197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147f00906ff143c7af29ec48dff92a78. Latest consensus state: current_term: 1 leader_uuid: "147f00906ff143c7af29ec48dff92a78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147f00906ff143c7af29ec48dff92a78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } } }
I20260619 06:44:44.548614 28197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:44.549137 28196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "147f00906ff143c7af29ec48dff92a78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147f00906ff143c7af29ec48dff92a78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39225 } } }
I20260619 06:44:44.549485 28196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:44.559092 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:44.542076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35559 (local address 127.0.0.1:39225)
0619 06:44:44.542238 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:44.542245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:44.542699 (+ 454us) server_negotiation.cc:407] Connection header received
0619 06:44:44.542899 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:44.542903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:44.542970 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:44.543057 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:44.544780 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:44.545745 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:44.547049 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:44.547726 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:44.547825 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:44.551217 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:44.551225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:44.551230 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:44.551274 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:44.555161 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:44.556189 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:44.556195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:44.556199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:44.556382 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:44.556960 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:44.556966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:44.556970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:44.557410 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:44.557663 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:44.558503 (+ 840us) server_negotiation.cc:299] Negotiation successful
0619 06:44:44.558693 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:44:44.559909 28200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:44.565239 28200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:44.573797 28200 catalog_manager.cc:1347] Generated new cluster ID: 36f758f652fe471c9fb073773fc76f03
I20260619 06:44:44.573937 28200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:44.582032 28200 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:44.583887 28200 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:44.595911 28200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 147f00906ff143c7af29ec48dff92a78: Generated new TSK 0
I20260619 06:44:44.596868 28200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:44.610045 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28118
I20260619 06:44:44.611545 28180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:44.702544 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 28118
W20260619 06:44:44.716130 28103 connection.cc:535] client connection to 127.0.0.1:39225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39225): Connection reset by peer (error 104)
2026-06-19T06:44:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1889 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:44 dist-test-slave-qfk4 krb5kdc[28218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:44 dist-test-slave-qfk4 krb5kdc[28218](info): set up 2 sockets
Jun 19 06:44:44 dist-test-slave-qfk4 krb5kdc[28218](info): commencing operation
krb5kdc: starting...
W20260619 06:44:45.720958 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:45 dist-test-slave-qfk4 krb5kdc[28218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:44:45.955562 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:46.280421 28231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:46.280761 28231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:46.280905 28231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:46.288466 28231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:46.288610 28231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:46.288697 28231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:46.288780 28231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:46.288838 28231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:46.298728 28231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:46.300030 28231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:46.301357 28231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:46.312696 28236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:46.312749 28239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:46.312772 28237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:46.313997 28231 server_base.cc:1034] running on GCE node
I20260619 06:44:46.314795 28231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:46.316265 28231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:46.317479 28231 hybrid_clock.cc:648] HybridClock initialized: now 1781851486317437 us; error 29 us; skew 500 ppm
I20260619 06:44:46.317816 28231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:46 dist-test-slave-qfk4 krb5kdc[28218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:46.324543 28231 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:44:46.326150 28231 webserver.cc:460] Webserver started at http://127.0.0.1:37061/ using document root <none> and password file <none>
I20260619 06:44:46.326927 28231 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:46.327060 28231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:46.327353 28231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:46.330477 28231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "9f3aa26ad5e444ffa9d56b2e8e2bff41"
format_stamp: "Formatted at 2026-06-19 06:44:46 on dist-test-slave-qfk4"
server_key: "6eb97ab9d8685c3ca20a4a8675a5e964"
server_key_iv: "20a0483ad55543010a6c8f70b6ca3565"
server_key_version: "encryptionkey@0"
I20260619 06:44:46.331254 28231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "9f3aa26ad5e444ffa9d56b2e8e2bff41"
format_stamp: "Formatted at 2026-06-19 06:44:46 on dist-test-slave-qfk4"
server_key: "6eb97ab9d8685c3ca20a4a8675a5e964"
server_key_iv: "20a0483ad55543010a6c8f70b6ca3565"
server_key_version: "encryptionkey@0"
I20260619 06:44:46.336503 28231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 06:44:46.340171 28246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:46.341624 28231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:44:46.341791 28231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "9f3aa26ad5e444ffa9d56b2e8e2bff41"
format_stamp: "Formatted at 2026-06-19 06:44:46 on dist-test-slave-qfk4"
server_key: "6eb97ab9d8685c3ca20a4a8675a5e964"
server_key_iv: "20a0483ad55543010a6c8f70b6ca3565"
server_key_version: "encryptionkey@0"
I20260619 06:44:46.341985 28231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:46.364615 28231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:46.368889 28231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:46.369167 28231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:46.386734 28231 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41863
I20260619 06:44:46.386757 28297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41863 every 8 connection(s)
I20260619 06:44:46.388135 28231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:46.392056 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:46.398212 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 28231
I20260619 06:44:46.398420 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:46.398964 7755 external_mini_cluster.cc:1428] Setting key 44935093f2427616882060ac5f8fc34e
I20260619 06:44:46.403934 28298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41: Bootstrap starting.
I20260619 06:44:46.407163 28298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:46.408195 28298 log.cc:824] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:46.411463 28298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41: No bootstrap required, opened a new log
I20260619 06:44:46.412546 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:46.401042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53769 (local address 127.0.0.1:41863)
0619 06:44:46.401920 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:46.401933 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:46.401987 (+ 54us) server_negotiation.cc:407] Connection header received
0619 06:44:46.403741 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:46.403772 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:46.404252 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:46.404648 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:46.405198 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:46.406524 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:46.407907 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:46.408706 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:46.408901 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:46.410161 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:46.410169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:46.410185 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:46.410714 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:46.410784 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:46.410814 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:46.410927 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:46.411254 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 06:44:46.411434 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":411,"threads_started":1}
I20260619 06:44:46.416651 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:44:46.419823 28298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3aa26ad5e444ffa9d56b2e8e2bff41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } }
I20260619 06:44:46.420212 28298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:46.420300 28298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3aa26ad5e444ffa9d56b2e8e2bff41, State: Initialized, Role: FOLLOWER
I20260619 06:44:46.421063 28298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3aa26ad5e444ffa9d56b2e8e2bff41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } }
I20260619 06:44:46.421350 28298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:46.421490 28298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:46.421670 28298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:46.423933 28298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3aa26ad5e444ffa9d56b2e8e2bff41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } }
I20260619 06:44:46.424434 28298 leader_election.cc:302] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f3aa26ad5e444ffa9d56b2e8e2bff41; no voters:
I20260619 06:44:46.425113 28298 leader_election.cc:288] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:46.425230 28308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:46.427220 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:46.420771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35769 (local address 127.0.0.1:41863)
0619 06:44:46.420917 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:46.420921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:46.421428 (+ 507us) server_negotiation.cc:407] Connection header received
0619 06:44:46.421639 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:46.421644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:46.421711 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:46.421790 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:46.422362 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:46.423162 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:46.424480 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:46.425044 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:46.425181 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:46.426432 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:46.426438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:46.426441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:46.426459 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:46.426530 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:46.426546 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:46.426639 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:46.426831 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 06:44:46.426894 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:44:46.427608 28298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:46.428829 28308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [term 1 LEADER]: Becoming Leader. State: Replica: 9f3aa26ad5e444ffa9d56b2e8e2bff41, State: Running, Role: LEADER
I20260619 06:44:46.429533 28308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3aa26ad5e444ffa9d56b2e8e2bff41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } }
I20260619 06:44:46.433596 28310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f3aa26ad5e444ffa9d56b2e8e2bff41. Latest consensus state: current_term: 1 leader_uuid: "9f3aa26ad5e444ffa9d56b2e8e2bff41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3aa26ad5e444ffa9d56b2e8e2bff41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } } }
I20260619 06:44:46.434157 28310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:46.433842 28309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f3aa26ad5e444ffa9d56b2e8e2bff41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3aa26ad5e444ffa9d56b2e8e2bff41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41863 } } }
I20260619 06:44:46.434825 28309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:46.441563 28320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:46.445917 28320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:46.454949 28320 catalog_manager.cc:1347] Generated new cluster ID: 00797ecfc10b41b5bfc44d730e7e762a
I20260619 06:44:46.455086 28320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:46.474712 28320 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:46.476778 28320 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:46.484325 28320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f3aa26ad5e444ffa9d56b2e8e2bff41: Generated new TSK 0
I20260619 06:44:46.485461 28320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:46.492141 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28231
W20260619 06:44:46.493247 28293 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53769
W20260619 06:44:46.493957 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28231: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:44:46.494130 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 28231
W20260619 06:44:46.508051 28216 connection.cc:535] client connection to 127.0.0.1:41863 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41863): Connection reset by peer (error 104)
2026-06-19T06:44:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1787 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:46 dist-test-slave-qfk4 krb5kdc[28329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:46 dist-test-slave-qfk4 krb5kdc[28329](info): set up 2 sockets
Jun 19 06:44:46 dist-test-slave-qfk4 krb5kdc[28329](info): commencing operation
krb5kdc: starting...
W20260619 06:44:47.508777 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:47 dist-test-slave-qfk4 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:44:47.743418 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:48.063160 28342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:48.063467 28342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:48.063598 28342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:48.070816 28342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:48.070945 28342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:48.071023 28342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:48.071099 28342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:48.071156 28342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:48.082084 28342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:48.083382 28342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:48.084718 28342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:48.097018 28347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:48.097044 28348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:48.097138 28350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:48.097842 28342 server_base.cc:1034] running on GCE node
I20260619 06:44:48.098630 28342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:48.100049 28342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:48.101253 28342 hybrid_clock.cc:648] HybridClock initialized: now 1781851488101198 us; error 47 us; skew 500 ppm
I20260619 06:44:48.101615 28342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:48 dist-test-slave-qfk4 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:48.104661 28342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:44:48.106048 28342 webserver.cc:460] Webserver started at http://127.0.0.1:33445/ using document root <none> and password file <none>
I20260619 06:44:48.106715 28342 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:48.106827 28342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:48.107102 28342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:48.110062 28342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "5eed79682ab34107bd6a3c46e9c146e4"
format_stamp: "Formatted at 2026-06-19 06:44:48 on dist-test-slave-qfk4"
server_key: "fc5d30d575be990e877679d0d206ef51"
server_key_iv: "a822b261853e8795af4948ff46aa47fc"
server_key_version: "encryptionkey@0"
I20260619 06:44:48.110795 28342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "5eed79682ab34107bd6a3c46e9c146e4"
format_stamp: "Formatted at 2026-06-19 06:44:48 on dist-test-slave-qfk4"
server_key: "fc5d30d575be990e877679d0d206ef51"
server_key_iv: "a822b261853e8795af4948ff46aa47fc"
server_key_version: "encryptionkey@0"
I20260619 06:44:48.115896 28342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:44:48.119508 28357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:48.121045 28342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:44:48.121212 28342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "5eed79682ab34107bd6a3c46e9c146e4"
format_stamp: "Formatted at 2026-06-19 06:44:48 on dist-test-slave-qfk4"
server_key: "fc5d30d575be990e877679d0d206ef51"
server_key_iv: "a822b261853e8795af4948ff46aa47fc"
server_key_version: "encryptionkey@0"
I20260619 06:44:48.121399 28342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:48.127348 28342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:48.131783 28342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:48.132054 28342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:48.149461 28342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40377
I20260619 06:44:48.149515 28408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40377 every 8 connection(s)
I20260619 06:44:48.150900 28342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:48.154654 28409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:48.155671 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 28342
I20260619 06:44:48.155812 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:48.156226 7755 external_mini_cluster.cc:1428] Setting key d6771aff5f94b324ad5c53faf82cc57b
I20260619 06:44:48.163810 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:48.158014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35517 (local address 127.0.0.1:40377)
0619 06:44:48.158786 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:48.158798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:48.158842 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:44:48.160470 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:48.160493 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:48.160910 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:48.161240 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:48.162271 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:48.162276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:48.162290 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:48.162303 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:48.162360 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:48.162405 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:48.162561 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:48.162699 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 06:44:48.162878 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":338,"threads_started":1}
I20260619 06:44:48.168093 28409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4: Bootstrap starting.
I20260619 06:44:48.168305 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:44:48.171885 28409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:48.173359 28409 log.cc:824] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:48.176721 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:48.174732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49399 (local address 127.0.0.1:40377)
0619 06:44:48.174932 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:48.174939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:48.174958 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:44:48.175011 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:48.175015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:48.175068 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:48.175149 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:48.176118 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:48.176122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:48.176125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:48.176140 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:48.176181 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:48.176197 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:48.176285 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:48.176390 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 06:44:48.176436 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 06:44:48.182507 28409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4: No bootstrap required, opened a new log
I20260619 06:44:48.189931 28409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed79682ab34107bd6a3c46e9c146e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } }
I20260619 06:44:48.190351 28409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:48.190444 28409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eed79682ab34107bd6a3c46e9c146e4, State: Initialized, Role: FOLLOWER
I20260619 06:44:48.191059 28409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed79682ab34107bd6a3c46e9c146e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } }
I20260619 06:44:48.191290 28409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:48.191414 28409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:48.191565 28409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:48.193827 28409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed79682ab34107bd6a3c46e9c146e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } }
I20260619 06:44:48.194332 28409 leader_election.cc:302] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eed79682ab34107bd6a3c46e9c146e4; no voters:
I20260619 06:44:48.194921 28409 leader_election.cc:288] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:48.195051 28419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:48.195684 28419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [term 1 LEADER]: Becoming Leader. State: Replica: 5eed79682ab34107bd6a3c46e9c146e4, State: Running, Role: LEADER
I20260619 06:44:48.196194 28419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed79682ab34107bd6a3c46e9c146e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } }
I20260619 06:44:48.197202 28409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:48.199339 28421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eed79682ab34107bd6a3c46e9c146e4. Latest consensus state: current_term: 1 leader_uuid: "5eed79682ab34107bd6a3c46e9c146e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed79682ab34107bd6a3c46e9c146e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } } }
I20260619 06:44:48.199715 28421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:48.199155 28420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eed79682ab34107bd6a3c46e9c146e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eed79682ab34107bd6a3c46e9c146e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } } }
I20260619 06:44:48.200809 28420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:48.201485 28424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:48.206907 28424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:48.216001 28424 catalog_manager.cc:1347] Generated new cluster ID: 3904556d14704684b9d96b9a2e85824e
I20260619 06:44:48.216120 28424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:48.240285 28424 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:48.240760 28424 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:48.248651 28424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eed79682ab34107bd6a3c46e9c146e4: Generated new TSK 0
I20260619 06:44:48.249477 28424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:48.262873 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28342
W20260619 06:44:48.264292 28404 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35517
W20260619 06:44:48.265008 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28342: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:44:48.265134 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 28342
2026-06-19T06:44:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1770 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:48 dist-test-slave-qfk4 krb5kdc[28440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:48 dist-test-slave-qfk4 krb5kdc[28440](info): set up 2 sockets
Jun 19 06:44:48 dist-test-slave-qfk4 krb5kdc[28440](info): commencing operation
krb5kdc: starting...
W20260619 06:44:49.272859 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:49 dist-test-slave-qfk4 krb5kdc[28440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:44:49.513373 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:49.848066 28453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:49.848388 28453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:49.848524 28453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:49.855808 28453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:49.855937 28453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:49.856016 28453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:49.856101 28453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:49.856159 28453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:49.866025 28453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:49.867393 28453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:49.868707 28453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:49.880342 28461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:49.880271 28458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:49.880301 28459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:49.881486 28453 server_base.cc:1034] running on GCE node
I20260619 06:44:49.882349 28453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:49.883886 28453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:49.885082 28453 hybrid_clock.cc:648] HybridClock initialized: now 1781851489885032 us; error 43 us; skew 500 ppm
I20260619 06:44:49.885416 28453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:49 dist-test-slave-qfk4 krb5kdc[28440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:49.888433 28453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:44:49.889921 28453 webserver.cc:460] Webserver started at http://127.0.0.1:37249/ using document root <none> and password file <none>
I20260619 06:44:49.890599 28453 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:49.890722 28453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:49.890980 28453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:49.894008 28453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "89e807a073ac4e6b8807d7c6b2d9d7b6"
format_stamp: "Formatted at 2026-06-19 06:44:49 on dist-test-slave-qfk4"
server_key: "d15862fdf04894d9318e414bc97d7f3c"
server_key_iv: "ed9e2de2e7f501c66aa58c360f7857aa"
server_key_version: "encryptionkey@0"
I20260619 06:44:49.894794 28453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "89e807a073ac4e6b8807d7c6b2d9d7b6"
format_stamp: "Formatted at 2026-06-19 06:44:49 on dist-test-slave-qfk4"
server_key: "d15862fdf04894d9318e414bc97d7f3c"
server_key_iv: "ed9e2de2e7f501c66aa58c360f7857aa"
server_key_version: "encryptionkey@0"
I20260619 06:44:49.899986 28453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:44:49.903438 28468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:49.904904 28453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:44:49.905056 28453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "89e807a073ac4e6b8807d7c6b2d9d7b6"
format_stamp: "Formatted at 2026-06-19 06:44:49 on dist-test-slave-qfk4"
server_key: "d15862fdf04894d9318e414bc97d7f3c"
server_key_iv: "ed9e2de2e7f501c66aa58c360f7857aa"
server_key_version: "encryptionkey@0"
I20260619 06:44:49.905230 28453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:49.923316 28453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:49.927456 28453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:49.927695 28453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:49.943960 28453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33345
I20260619 06:44:49.943979 28519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33345 every 8 connection(s)
I20260619 06:44:49.945343 28453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:49.945729 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 28453
I20260619 06:44:49.945896 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:49.946470 7755 external_mini_cluster.cc:1428] Setting key fb7248d7da62bef31ba46b61e3575516
I20260619 06:44:49.949982 28520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:44:49 dist-test-slave-qfk4 krb5kdc[28440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:44:49.963867 28520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6: Bootstrap starting.
I20260619 06:44:49.968042 28520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:49.969336 28520 log.cc:824] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:49.972914 28520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6: No bootstrap required, opened a new log
I20260619 06:44:49.973170 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:49.948298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40475 (local address 127.0.0.1:33345)
0619 06:44:49.949052 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:49.949068 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:49.949120 (+ 52us) server_negotiation.cc:407] Connection header received
0619 06:44:49.951318 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:49.951346 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:49.951918 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:49.952353 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:49.953788 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:49.955839 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:49.957401 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:49.958424 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:49.958616 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:49.964297 (+ 5681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:49.964309 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:49.964331 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:49.964380 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:49.968357 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:49.969249 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:49.969258 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:49.969267 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:49.969519 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:49.969934 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:49.969941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:49.969945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:49.970588 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:49.970847 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:49.971357 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 06:44:49.971805 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":322,"threads_started":1}
I20260619 06:44:49.977018 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:44:49.982411 28520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e807a073ac4e6b8807d7c6b2d9d7b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33345 } }
I20260619 06:44:49.982789 28520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:49.982884 28520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e807a073ac4e6b8807d7c6b2d9d7b6, State: Initialized, Role: FOLLOWER
I20260619 06:44:49.983536 28520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e807a073ac4e6b8807d7c6b2d9d7b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33345 } }
I20260619 06:44:49.983755 28520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:49.983880 28520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:49.984038 28520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:49.986780 28520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e807a073ac4e6b8807d7c6b2d9d7b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33345 } }
I20260619 06:44:49.987309 28520 leader_election.cc:302] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e807a073ac4e6b8807d7c6b2d9d7b6; no voters:
I20260619 06:44:49.987964 28520 leader_election.cc:288] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:49.988061 28530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:49.988781 28530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [term 1 LEADER]: Becoming Leader. State: Replica: 89e807a073ac4e6b8807d7c6b2d9d7b6, State: Running, Role: LEADER
I20260619 06:44:49.989411 28530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e807a073ac4e6b8807d7c6b2d9d7b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33345 } }
I20260619 06:44:49.990314 28520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:49.991400 28531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89e807a073ac4e6b8807d7c6b2d9d7b6. Latest consensus state: current_term: 1 leader_uuid: "89e807a073ac4e6b8807d7c6b2d9d7b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e807a073ac4e6b8807d7c6b2d9d7b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33345 } } }
I20260619 06:44:49.991815 28531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:49.998273 28532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89e807a073ac4e6b8807d7c6b2d9d7b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e807a073ac4e6b8807d7c6b2d9d7b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33345 } } }
I20260619 06:44:49.998652 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:49.982205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56855 (local address 127.0.0.1:33345)
0619 06:44:49.982374 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:49.982380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:49.982805 (+ 425us) server_negotiation.cc:407] Connection header received
0619 06:44:49.983060 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:49.983065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:49.983141 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:49.983239 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:49.984781 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:49.985880 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:49.987109 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:49.987612 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:49.987727 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:49.990526 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:49.990533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:49.990538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:49.990586 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:49.994240 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:49.995191 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:49.995197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:49.995201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:49.995431 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:49.995905 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:49.995910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:49.995914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:49.996283 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:49.997793 (+ 1510us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:49.997946 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 06:44:49.998178 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:44:49.999938 28532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:50.001585 28535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:50.006950 28535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:50.018157 28535 catalog_manager.cc:1347] Generated new cluster ID: f8c018cc3dda4f03bcda76a5daca3f48
I20260619 06:44:50.018286 28535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:50.038372 28535 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:50.039755 28535 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:50.050912 28535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89e807a073ac4e6b8807d7c6b2d9d7b6: Generated new TSK 0
I20260619 06:44:50.051874 28535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:50.058977 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28453
I20260619 06:44:50.060178 28515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:50.153156 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 28453
2026-06-19T06:44:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1889 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:50 dist-test-slave-qfk4 krb5kdc[28552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:50 dist-test-slave-qfk4 krb5kdc[28552](info): set up 2 sockets
Jun 19 06:44:50 dist-test-slave-qfk4 krb5kdc[28552](info): commencing operation
krb5kdc: starting...
W20260619 06:44:51.168774 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:51 dist-test-slave-qfk4 krb5kdc[28552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:44:51.407835 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:51.729398 28565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:51.729701 28565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:51.729839 28565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:51.737124 28565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:51.737257 28565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:51.737336 28565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:51.737418 28565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:51.737488 28565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:51.747545 28565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:51.748872 28565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:51.750193 28565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:51.761969 28570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:51.761938 28571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:51.761909 28573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:51.762934 28565 server_base.cc:1034] running on GCE node
I20260619 06:44:51.763758 28565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:51.765275 28565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:51.766471 28565 hybrid_clock.cc:648] HybridClock initialized: now 1781851491766417 us; error 48 us; skew 500 ppm
I20260619 06:44:51.766842 28565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:51 dist-test-slave-qfk4 krb5kdc[28552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:51.773485 28565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:44:51.775264 28565 webserver.cc:460] Webserver started at http://127.0.0.1:39191/ using document root <none> and password file <none>
I20260619 06:44:51.776036 28565 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:51.776165 28565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:51.776531 28565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:51.779666 28565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "04b87c178f7046989b16c9dcc42b7aae"
format_stamp: "Formatted at 2026-06-19 06:44:51 on dist-test-slave-qfk4"
server_key: "c3f8e22e6c29dc704e6d2f1993e36aaa"
server_key_iv: "682656031a171adc79e2802de8073b73"
server_key_version: "encryptionkey@0"
I20260619 06:44:51.780431 28565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "04b87c178f7046989b16c9dcc42b7aae"
format_stamp: "Formatted at 2026-06-19 06:44:51 on dist-test-slave-qfk4"
server_key: "c3f8e22e6c29dc704e6d2f1993e36aaa"
server_key_iv: "682656031a171adc79e2802de8073b73"
server_key_version: "encryptionkey@0"
I20260619 06:44:51.785972 28565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:44:51.789912 28580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:51.791383 28565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:44:51.791558 28565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "04b87c178f7046989b16c9dcc42b7aae"
format_stamp: "Formatted at 2026-06-19 06:44:51 on dist-test-slave-qfk4"
server_key: "c3f8e22e6c29dc704e6d2f1993e36aaa"
server_key_iv: "682656031a171adc79e2802de8073b73"
server_key_version: "encryptionkey@0"
I20260619 06:44:51.791751 28565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:51.823248 28565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:51.827431 28565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:51.827716 28565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:51.845074 28565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43177
I20260619 06:44:51.845093 28631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43177 every 8 connection(s)
I20260619 06:44:51.846448 28565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:51.850304 28632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:51.851070 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 28565
I20260619 06:44:51.851220 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:51.851632 7755 external_mini_cluster.cc:1428] Setting key e9d2c8044603f65a64470533b9c94080
I20260619 06:44:51.863930 28632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae: Bootstrap starting.
I20260619 06:44:51.865202 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:51.853420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60175 (local address 127.0.0.1:43177)
0619 06:44:51.854202 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:51.854213 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:51.854258 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:44:51.856217 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:51.856241 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:51.856735 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:51.857098 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:51.857703 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:51.858980 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:51.860884 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:51.861623 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:51.861754 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:51.862949 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:51.862956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:51.862970 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:51.863504 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:51.863592 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:51.863616 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:51.863700 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:51.863957 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 06:44:51.864204 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":348,"threads_started":1}
I20260619 06:44:51.867204 28632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:51.868309 28632 log.cc:824] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:51.868754 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:44:51.871348 28632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae: No bootstrap required, opened a new log
I20260619 06:44:51.880136 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:51.873697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57821 (local address 127.0.0.1:43177)
0619 06:44:51.873883 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:51.873889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:51.874252 (+ 363us) server_negotiation.cc:407] Connection header received
0619 06:44:51.874402 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:51.874408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:51.874489 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:51.874609 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:51.875099 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:51.876171 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:51.877597 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:51.878290 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:51.878407 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:51.879425 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:51.879432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:51.879437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:51.879459 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:51.879539 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:51.879563 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:51.879658 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:51.879729 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 06:44:51.879822 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:44:51.880860 28632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b87c178f7046989b16c9dcc42b7aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } }
I20260619 06:44:51.881356 28632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:51.881470 28632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b87c178f7046989b16c9dcc42b7aae, State: Initialized, Role: FOLLOWER
I20260619 06:44:51.882073 28632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b87c178f7046989b16c9dcc42b7aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } }
I20260619 06:44:51.882292 28632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:51.882416 28632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:51.882556 28632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:51.884725 28632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b87c178f7046989b16c9dcc42b7aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } }
I20260619 06:44:51.885200 28632 leader_election.cc:302] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04b87c178f7046989b16c9dcc42b7aae; no voters:
I20260619 06:44:51.885727 28632 leader_election.cc:288] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:51.885880 28642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:51.886672 28642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [term 1 LEADER]: Becoming Leader. State: Replica: 04b87c178f7046989b16c9dcc42b7aae, State: Running, Role: LEADER
I20260619 06:44:51.887280 28642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b87c178f7046989b16c9dcc42b7aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } }
I20260619 06:44:51.887844 28632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:51.890273 28644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04b87c178f7046989b16c9dcc42b7aae. Latest consensus state: current_term: 1 leader_uuid: "04b87c178f7046989b16c9dcc42b7aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b87c178f7046989b16c9dcc42b7aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } } }
I20260619 06:44:51.890718 28644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:51.891283 28643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04b87c178f7046989b16c9dcc42b7aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b87c178f7046989b16c9dcc42b7aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43177 } } }
I20260619 06:44:51.891563 28643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:51.893327 28651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:51.898291 28651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:51.907971 28651 catalog_manager.cc:1347] Generated new cluster ID: f996a937bc6c4c5289ee9ea5e3e30a97
I20260619 06:44:51.908105 28651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:51.931945 28651 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:51.933936 28651 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:51.940937 28651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04b87c178f7046989b16c9dcc42b7aae: Generated new TSK 0
I20260619 06:44:51.941990 28651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:51.952759 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28565
W20260619 06:44:51.953874 28627 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60175
W20260619 06:44:51.954558 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28565: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:44:51.954741 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 28565
2026-06-19T06:44:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1800 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:52 dist-test-slave-qfk4 krb5kdc[28663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:52 dist-test-slave-qfk4 krb5kdc[28663](info): set up 2 sockets
Jun 19 06:44:52 dist-test-slave-qfk4 krb5kdc[28663](info): commencing operation
krb5kdc: starting...
W20260619 06:44:52.976899 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:53 dist-test-slave-qfk4 krb5kdc[28663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:44:53.217036 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38123
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:53.539997 28676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:53.540306 28676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:53.540488 28676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:53.548161 28676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:53.548269 28676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:53.548337 28676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:53.548417 28676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:53.548476 28676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:53.558534 28676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:53.559809 28676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:53.561158 28676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:53.572320 28681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:53.572348 28684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:53.572393 28682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:53.572990 28676 server_base.cc:1034] running on GCE node
I20260619 06:44:53.573817 28676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:53.575187 28676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:53.576390 28676 hybrid_clock.cc:648] HybridClock initialized: now 1781851493576351 us; error 42 us; skew 500 ppm
I20260619 06:44:53.576747 28676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:53 dist-test-slave-qfk4 krb5kdc[28663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:53.579799 28676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:44:53.581306 28676 webserver.cc:460] Webserver started at http://127.0.0.1:40315/ using document root <none> and password file <none>
I20260619 06:44:53.581995 28676 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:53.582126 28676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:53.582413 28676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:53.585450 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "a159010d96a04ba7bbade3ac84172949"
format_stamp: "Formatted at 2026-06-19 06:44:53 on dist-test-slave-qfk4"
server_key: "a7bead1178e16833f637af2d91247ed8"
server_key_iv: "1387e43b01f7197b463ccbdffc053df1"
server_key_version: "encryptionkey@0"
I20260619 06:44:53.586236 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "a159010d96a04ba7bbade3ac84172949"
format_stamp: "Formatted at 2026-06-19 06:44:53 on dist-test-slave-qfk4"
server_key: "a7bead1178e16833f637af2d91247ed8"
server_key_iv: "1387e43b01f7197b463ccbdffc053df1"
server_key_version: "encryptionkey@0"
I20260619 06:44:53.591620 28676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:44:53.595348 28691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:53.596829 28676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:44:53.597044 28676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "a159010d96a04ba7bbade3ac84172949"
format_stamp: "Formatted at 2026-06-19 06:44:53 on dist-test-slave-qfk4"
server_key: "a7bead1178e16833f637af2d91247ed8"
server_key_iv: "1387e43b01f7197b463ccbdffc053df1"
server_key_version: "encryptionkey@0"
I20260619 06:44:53.597244 28676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:53.603107 28676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:53.607358 28676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:53.607656 28676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:53.625329 28676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38123
I20260619 06:44:53.625401 28742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38123 every 8 connection(s)
I20260619 06:44:53.626720 28676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:53.629398 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 28676
I20260619 06:44:53.629554 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:53.629981 7755 external_mini_cluster.cc:1428] Setting key 8d94873b52cb4219dc1d8507bb0e54f2
I20260619 06:44:53.630458 28743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:53.638293 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:53.631687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41271 (local address 127.0.0.1:38123)
0619 06:44:53.632414 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:53.632425 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:53.632469 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:44:53.634888 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:53.634912 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:53.635307 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:53.635681 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:53.636732 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:53.636738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:53.636751 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:53.636766 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:53.636828 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:53.636853 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:53.637031 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:53.637128 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 06:44:53.637310 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":327,"threads_started":1}
I20260619 06:44:53.641940 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:44:53.643926 28743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949: Bootstrap starting.
I20260619 06:44:53.647836 28743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:53.649200 28743 log.cc:824] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:53.649399 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:53.647072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48563 (local address 127.0.0.1:38123)
0619 06:44:53.647256 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:53.647261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:53.647557 (+ 296us) server_negotiation.cc:407] Connection header received
0619 06:44:53.647757 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:53.647760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:53.647808 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:53.647883 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:53.648781 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:53.648785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:53.648788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:53.648802 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:53.648842 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:53.648858 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:53.648946 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:53.649085 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 06:44:53.649130 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:44:53.652070 28743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949: No bootstrap required, opened a new log
I20260619 06:44:53.659243 28743 raft_consensus.cc:348] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159010d96a04ba7bbade3ac84172949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } }
I20260619 06:44:53.659655 28743 raft_consensus.cc:374] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:53.659746 28743 raft_consensus.cc:729] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a159010d96a04ba7bbade3ac84172949, State: Initialized, Role: FOLLOWER
I20260619 06:44:53.660281 28743 consensus_queue.cc:260] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159010d96a04ba7bbade3ac84172949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } }
I20260619 06:44:53.660496 28743 raft_consensus.cc:388] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:53.660686 28743 raft_consensus.cc:482] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:53.660833 28743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:53.663012 28743 raft_consensus.cc:504] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159010d96a04ba7bbade3ac84172949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } }
I20260619 06:44:53.663486 28743 leader_election.cc:302] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a159010d96a04ba7bbade3ac84172949; no voters:
I20260619 06:44:53.664006 28743 leader_election.cc:288] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:53.664145 28753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:53.664844 28753 raft_consensus.cc:686] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [term 1 LEADER]: Becoming Leader. State: Replica: a159010d96a04ba7bbade3ac84172949, State: Running, Role: LEADER
I20260619 06:44:53.665352 28753 consensus_queue.cc:237] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159010d96a04ba7bbade3ac84172949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } }
I20260619 06:44:53.665998 28743 sys_catalog.cc:564] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:53.667934 28755 sys_catalog.cc:455] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a159010d96a04ba7bbade3ac84172949. Latest consensus state: current_term: 1 leader_uuid: "a159010d96a04ba7bbade3ac84172949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159010d96a04ba7bbade3ac84172949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } } }
I20260619 06:44:53.668330 28755 sys_catalog.cc:458] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:53.673166 28754 sys_catalog.cc:455] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a159010d96a04ba7bbade3ac84172949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159010d96a04ba7bbade3ac84172949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38123 } } }
I20260619 06:44:53.673512 28754 sys_catalog.cc:458] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:53.675228 28759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:53.680841 28759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:53.689949 28759 catalog_manager.cc:1347] Generated new cluster ID: e44f43e7bb4744189e4c4b37e9f51534
I20260619 06:44:53.690062 28759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:53.711256 28759 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:53.711782 28759 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:53.723974 28759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a159010d96a04ba7bbade3ac84172949: Generated new TSK 0
I20260619 06:44:53.724851 28759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:53.737588 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28676
W20260619 06:44:53.739046 28738 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41271
W20260619 06:44:53.739777 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:44:53.739907 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 28676
2026-06-19T06:44:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1786 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:53 dist-test-slave-qfk4 krb5kdc[28774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:53 dist-test-slave-qfk4 krb5kdc[28774](info): set up 2 sockets
Jun 19 06:44:53 dist-test-slave-qfk4 krb5kdc[28774](info): commencing operation
krb5kdc: starting...
W20260619 06:44:54.756937 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:54 dist-test-slave-qfk4 krb5kdc[28774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:44:54.999307 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:55.322208 28787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:55.322503 28787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:55.322614 28787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:55.329846 28787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:55.329958 28787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:55.330030 28787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:55.330123 28787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:55.330188 28787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:55.339933 28787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:55.341250 28787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:55.342521 28787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:55.353763 28792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:55.353809 28793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:55.353832 28795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:55.354624 28787 server_base.cc:1034] running on GCE node
I20260619 06:44:55.355324 28787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:55.356721 28787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:55.357924 28787 hybrid_clock.cc:648] HybridClock initialized: now 1781851495357869 us; error 52 us; skew 500 ppm
I20260619 06:44:55.358268 28787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:55 dist-test-slave-qfk4 krb5kdc[28774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:55.361418 28787 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:44:55.363006 28787 webserver.cc:460] Webserver started at http://127.0.0.1:45145/ using document root <none> and password file <none>
I20260619 06:44:55.363709 28787 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:55.363823 28787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:55.364101 28787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:55.367089 28787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "972d0dc0be6b45b7ac72899e4c4caaf2"
format_stamp: "Formatted at 2026-06-19 06:44:55 on dist-test-slave-qfk4"
server_key: "9fc9404618cd0bef47bbc5c82d06ccf1"
server_key_iv: "a3fde43874c8ba9715643836e0995f86"
server_key_version: "encryptionkey@0"
I20260619 06:44:55.367812 28787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "972d0dc0be6b45b7ac72899e4c4caaf2"
format_stamp: "Formatted at 2026-06-19 06:44:55 on dist-test-slave-qfk4"
server_key: "9fc9404618cd0bef47bbc5c82d06ccf1"
server_key_iv: "a3fde43874c8ba9715643836e0995f86"
server_key_version: "encryptionkey@0"
I20260619 06:44:55.373171 28787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:44:55.376489 28802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:55.377851 28787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:44:55.378015 28787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "972d0dc0be6b45b7ac72899e4c4caaf2"
format_stamp: "Formatted at 2026-06-19 06:44:55 on dist-test-slave-qfk4"
server_key: "9fc9404618cd0bef47bbc5c82d06ccf1"
server_key_iv: "a3fde43874c8ba9715643836e0995f86"
server_key_version: "encryptionkey@0"
I20260619 06:44:55.378191 28787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:55.398212 28787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:55.402518 28787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:55.402793 28787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:55.420274 28787 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36245
I20260619 06:44:55.420318 28853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36245 every 8 connection(s)
I20260619 06:44:55.421732 28787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:55.425447 28854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:55.432067 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 28787
I20260619 06:44:55.432224 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:55.432663 7755 external_mini_cluster.cc:1428] Setting key b5e36a6c32e721c56d91efe2072ce6db
I20260619 06:44:55.436112 28854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2: Bootstrap starting.
I20260619 06:44:55.439877 28854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:55.441256 28854 log.cc:824] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:55.444312 28854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2: No bootstrap required, opened a new log
Jun 19 06:44:55 dist-test-slave-qfk4 krb5kdc[28774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:44:55.453155 28854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d0dc0be6b45b7ac72899e4c4caaf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260619 06:44:55.453583 28854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:55.453694 28854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972d0dc0be6b45b7ac72899e4c4caaf2, State: Initialized, Role: FOLLOWER
I20260619 06:44:55.454337 28854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d0dc0be6b45b7ac72899e4c4caaf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260619 06:44:55.454568 28854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:55.454702 28854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:55.454878 28854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:55.456863 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:55.434430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:42631 (local address 127.0.0.1:36245)
0619 06:44:55.435203 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:55.435218 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:55.435265 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:44:55.437689 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:55.437724 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:55.438301 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:55.438749 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:55.440062 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:55.442040 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:55.443473 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:55.444479 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:55.444711 (+ 232us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:55.448965 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:55.448973 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:55.448989 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:55.449046 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:55.452206 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:55.453281 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:55.453288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:55.453294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:55.453465 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:55.453965 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:55.453969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:55.453971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:55.454594 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:55.454906 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:55.455395 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 06:44:55.455738 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":304,"threads_started":1}
I20260619 06:44:55.457425 28854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d0dc0be6b45b7ac72899e4c4caaf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260619 06:44:55.457959 28854 leader_election.cc:302] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 972d0dc0be6b45b7ac72899e4c4caaf2; no voters:
I20260619 06:44:55.458550 28854 leader_election.cc:288] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:55.458695 28859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:55.459306 28859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [term 1 LEADER]: Becoming Leader. State: Replica: 972d0dc0be6b45b7ac72899e4c4caaf2, State: Running, Role: LEADER
I20260619 06:44:55.460000 28859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d0dc0be6b45b7ac72899e4c4caaf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260619 06:44:55.461402 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:44:55.461511 28854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:55.468029 28861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "972d0dc0be6b45b7ac72899e4c4caaf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d0dc0be6b45b7ac72899e4c4caaf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } } }
I20260619 06:44:55.468459 28861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:55.469323 28876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:55.469177 28860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 972d0dc0be6b45b7ac72899e4c4caaf2. Latest consensus state: current_term: 1 leader_uuid: "972d0dc0be6b45b7ac72899e4c4caaf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d0dc0be6b45b7ac72899e4c4caaf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } } }
I20260619 06:44:55.469497 28860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:55.474666 28876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:55.487012 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:55.467418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:46141 (local address 127.0.0.1:36245)
0619 06:44:55.467602 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:55.467608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:55.468148 (+ 540us) server_negotiation.cc:407] Connection header received
0619 06:44:55.468323 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:55.468329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:55.468403 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:55.468506 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:55.470729 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:55.472007 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:55.473996 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:55.474768 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:55.474896 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:55.478461 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:55.478468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:55.478472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:44:55.478515 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:55.482180 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:55.483369 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:55.483375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:55.483379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:55.483550 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:44:55.484185 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:44:55.484190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:44:55.484194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:44:55.484652 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:55.486187 (+ 1535us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:55.486379 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 06:44:55.486603 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:44:55.491843 28876 catalog_manager.cc:1347] Generated new cluster ID: f71fe08efc794107a9535064b41a77ae
I20260619 06:44:55.491967 28876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:55.522383 28876 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:55.524426 28876 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:55.536851 28876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 972d0dc0be6b45b7ac72899e4c4caaf2: Generated new TSK 0
I20260619 06:44:55.537951 28876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:55.544761 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28787
I20260619 06:44:55.546160 28849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:44:55.640089 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 28787
W20260619 06:44:55.653744 28772 connection.cc:535] client connection to 127.0.0.1:36245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36245): Connection reset by peer (error 104)
2026-06-19T06:44:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1899 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:55 dist-test-slave-qfk4 krb5kdc[28886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:55 dist-test-slave-qfk4 krb5kdc[28886](info): set up 2 sockets
Jun 19 06:44:55 dist-test-slave-qfk4 krb5kdc[28886](info): commencing operation
krb5kdc: starting...
W20260619 06:44:56.664924 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.011s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:56 dist-test-slave-qfk4 krb5kdc[28886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:44:56.903617 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:57.229518 28899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:57.229835 28899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:57.229957 28899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:57.237686 28899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:57.237814 28899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:57.237859 28899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:57.237920 28899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:57.237962 28899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:57.248029 28899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:57.249322 28899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:57.250629 28899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:57.262281 28904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:57.262306 28905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:57.262480 28907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:57.263144 28899 server_base.cc:1034] running on GCE node
I20260619 06:44:57.263979 28899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:57.265553 28899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:57.266760 28899 hybrid_clock.cc:648] HybridClock initialized: now 1781851497266719 us; error 32 us; skew 500 ppm
I20260619 06:44:57.267108 28899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:57 dist-test-slave-qfk4 krb5kdc[28886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:57.273257 28899 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:44:57.274744 28899 webserver.cc:460] Webserver started at http://127.0.0.1:42587/ using document root <none> and password file <none>
I20260619 06:44:57.275447 28899 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:57.275573 28899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:57.275858 28899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:57.279049 28899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "8d06452b662f4480a22185a48aa9d3cf"
format_stamp: "Formatted at 2026-06-19 06:44:57 on dist-test-slave-qfk4"
server_key: "284e7b4bfeb6ffb7c317164810fd6fc1"
server_key_iv: "1d464d775402e449c008d70721ef74d5"
server_key_version: "encryptionkey@0"
I20260619 06:44:57.279839 28899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "8d06452b662f4480a22185a48aa9d3cf"
format_stamp: "Formatted at 2026-06-19 06:44:57 on dist-test-slave-qfk4"
server_key: "284e7b4bfeb6ffb7c317164810fd6fc1"
server_key_iv: "1d464d775402e449c008d70721ef74d5"
server_key_version: "encryptionkey@0"
I20260619 06:44:57.285223 28899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 06:44:57.288782 28914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:57.290139 28899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:44:57.290310 28899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "8d06452b662f4480a22185a48aa9d3cf"
format_stamp: "Formatted at 2026-06-19 06:44:57 on dist-test-slave-qfk4"
server_key: "284e7b4bfeb6ffb7c317164810fd6fc1"
server_key_iv: "1d464d775402e449c008d70721ef74d5"
server_key_version: "encryptionkey@0"
I20260619 06:44:57.290500 28899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:57.308079 28899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:57.312376 28899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:57.312676 28899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:57.329167 28899 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40359
I20260619 06:44:57.329195 28965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40359 every 8 connection(s)
I20260619 06:44:57.330485 28899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:57.334184 28966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:57.337586 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 28899
I20260619 06:44:57.337764 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:57.338189 7755 external_mini_cluster.cc:1428] Setting key 02645161d49cd59de93d3c623ad745eb
I20260619 06:44:57.347404 28966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf: Bootstrap starting.
I20260619 06:44:57.351214 28966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:57.352546 28966 log.cc:824] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:57.352847 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:57.340065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:37513 (local address 127.0.0.1:40359)
0619 06:44:57.340848 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:57.340862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:57.340913 (+ 51us) server_negotiation.cc:407] Connection header received
0619 06:44:57.342965 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:57.342996 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:57.343585 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:57.344034 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:57.344579 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:57.346513 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:57.347829 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:57.348833 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:57.349041 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:57.350098 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:57.350108 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:57.350124 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:57.350810 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:57.350906 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:57.350938 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:57.351085 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:57.351299 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 06:44:57.351483 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":287,"threads_started":1}
I20260619 06:44:57.355739 28966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf: No bootstrap required, opened a new log
I20260619 06:44:57.356992 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:44:57.364280 28966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06452b662f4480a22185a48aa9d3cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } }
I20260619 06:44:57.364815 28966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:57.364938 28966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d06452b662f4480a22185a48aa9d3cf, State: Initialized, Role: FOLLOWER
I20260619 06:44:57.365554 28966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06452b662f4480a22185a48aa9d3cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } }
I20260619 06:44:57.365830 28966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:57.365973 28966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:57.366159 28966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:57.367535 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:57.360976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:45281 (local address 127.0.0.1:40359)
0619 06:44:57.361169 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:57.361176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:57.361572 (+ 396us) server_negotiation.cc:407] Connection header received
0619 06:44:57.361750 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:57.361754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:57.361812 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:57.361889 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:57.362444 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:57.363215 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:57.364999 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:44:57.365566 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:44:57.365656 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:44:57.366823 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:57.366827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:57.366830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:57.366851 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:57.366904 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:57.366919 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:57.367009 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:57.367170 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 06:44:57.367214 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:44:57.368943 28966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06452b662f4480a22185a48aa9d3cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } }
I20260619 06:44:57.369408 28966 leader_election.cc:302] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d06452b662f4480a22185a48aa9d3cf; no voters:
I20260619 06:44:57.369921 28966 leader_election.cc:288] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:57.370044 28976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:57.370714 28976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [term 1 LEADER]: Becoming Leader. State: Replica: 8d06452b662f4480a22185a48aa9d3cf, State: Running, Role: LEADER
I20260619 06:44:57.371266 28976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06452b662f4480a22185a48aa9d3cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } }
I20260619 06:44:57.371973 28966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:57.373765 28978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d06452b662f4480a22185a48aa9d3cf. Latest consensus state: current_term: 1 leader_uuid: "8d06452b662f4480a22185a48aa9d3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06452b662f4480a22185a48aa9d3cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } } }
I20260619 06:44:57.373718 28977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d06452b662f4480a22185a48aa9d3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d06452b662f4480a22185a48aa9d3cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } } }
I20260619 06:44:57.374142 28978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:57.374143 28977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:57.374917 28981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:57.379946 28981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:57.388517 28981 catalog_manager.cc:1347] Generated new cluster ID: 29c7a574ffb24a6a9edb710903e73352
I20260619 06:44:57.388676 28981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:57.407402 28981 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:57.408876 28981 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:57.423475 28981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d06452b662f4480a22185a48aa9d3cf: Generated new TSK 0
I20260619 06:44:57.424350 28981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:57.430821 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28899
W20260619 06:44:57.432413 28961 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.156:37513
W20260619 06:44:57.433133 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 28899: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:44:57.433260 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 28899
W20260619 06:44:57.446623 28884 connection.cc:535] client connection to 127.0.0.1:40359 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40359): Connection reset by peer (error 104)
2026-06-19T06:44:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1792 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:57 dist-test-slave-qfk4 krb5kdc[28997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:57 dist-test-slave-qfk4 krb5kdc[28997](info): set up 2 sockets
Jun 19 06:44:57 dist-test-slave-qfk4 krb5kdc[28997](info): commencing operation
krb5kdc: starting...
W20260619 06:44:58.445032 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:44:58 dist-test-slave-qfk4 krb5kdc[28997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:44:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:44:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:44:58.688011 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:44:59.018445 29010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:44:59.018759 29010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:44:59.018850 29010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:44:59.026296 29010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:44:59.026427 29010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:44:59.026472 29010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:44:59.026528 29010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:44:59.026573 29010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:44:59.036691 29010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:44:59.037932 29010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:44:59.039189 29010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:44:59.050849 29015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:59.050877 29018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:44:59.050815 29016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:44:59.052263 29010 server_base.cc:1034] running on GCE node
I20260619 06:44:59.053100 29010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:44:59.054539 29010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:44:59.055755 29010 hybrid_clock.cc:648] HybridClock initialized: now 1781851499055702 us; error 48 us; skew 500 ppm
I20260619 06:44:59.056128 29010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:44:59 dist-test-slave-qfk4 krb5kdc[28997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:44:59.059258 29010 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:44:59.060871 29010 webserver.cc:460] Webserver started at http://127.0.0.1:45041/ using document root <none> and password file <none>
I20260619 06:44:59.061570 29010 fs_manager.cc:359] Metadata directory not provided
I20260619 06:44:59.061687 29010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:44:59.061973 29010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:44:59.064998 29010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "f81ebb9d2ecc4a15be34f0d8370841f2"
format_stamp: "Formatted at 2026-06-19 06:44:59 on dist-test-slave-qfk4"
server_key: "5c9d76cb072cb4710f54c81b236ddcd7"
server_key_iv: "88773b36f09d124cad0f658ca477520b"
server_key_version: "encryptionkey@0"
I20260619 06:44:59.065766 29010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "f81ebb9d2ecc4a15be34f0d8370841f2"
format_stamp: "Formatted at 2026-06-19 06:44:59 on dist-test-slave-qfk4"
server_key: "5c9d76cb072cb4710f54c81b236ddcd7"
server_key_iv: "88773b36f09d124cad0f658ca477520b"
server_key_version: "encryptionkey@0"
I20260619 06:44:59.070884 29010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 06:44:59.074381 29025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:44:59.075703 29010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:44:59.075865 29010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "f81ebb9d2ecc4a15be34f0d8370841f2"
format_stamp: "Formatted at 2026-06-19 06:44:59 on dist-test-slave-qfk4"
server_key: "5c9d76cb072cb4710f54c81b236ddcd7"
server_key_iv: "88773b36f09d124cad0f658ca477520b"
server_key_version: "encryptionkey@0"
I20260619 06:44:59.076046 29010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:44:59.081621 29010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:44:59.085747 29010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:44:59.086017 29010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:44:59.103410 29010 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37619
I20260619 06:44:59.103456 29076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37619 every 8 connection(s)
I20260619 06:44:59.104892 29010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:44:59.108584 29077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:44:59.112380 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 29010
I20260619 06:44:59.112557 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:44:59.113018 7755 external_mini_cluster.cc:1428] Setting key 76b75ce12d069e5b257ee2310947f6fd
I20260619 06:44:59.120970 29077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2: Bootstrap starting.
I20260619 06:44:59.123241 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:59.115703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:60831 (local address 127.0.0.1:37619)
0619 06:44:59.117662 (+ 1959us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:59.117680 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:59.118021 (+ 341us) server_negotiation.cc:407] Connection header received
0619 06:44:59.119888 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:59.119912 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:59.120294 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:59.120689 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:59.121713 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:59.121719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:59.121733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:59.121747 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:59.121802 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:59.121824 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:59.121983 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:59.122108 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 06:44:59.122300 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1660,"thread_start_us":1335,"threads_started":1}
I20260619 06:44:59.124357 29077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2: Neither blocks nor log segments found. Creating new log.
I20260619 06:44:59.125449 29077 log.cc:824] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:44:59.126950 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:44:59.128296 29077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2: No bootstrap required, opened a new log
I20260619 06:44:59.133651 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:44:59.131179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:53171 (local address 127.0.0.1:37619)
0619 06:44:59.131372 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:44:59.131378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:44:59.131677 (+ 299us) server_negotiation.cc:407] Connection header received
0619 06:44:59.131898 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:44:59.131901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:44:59.131951 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:44:59.132058 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:44:59.133045 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:44:59.133049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:44:59.133052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:44:59.133068 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:44:59.133109 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:44:59.133124 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:44:59.133217 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:44:59.133330 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 06:44:59.133380 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:44:59.137152 29077 raft_consensus.cc:348] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81ebb9d2ecc4a15be34f0d8370841f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } }
I20260619 06:44:59.137531 29077 raft_consensus.cc:374] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:44:59.137626 29077 raft_consensus.cc:729] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81ebb9d2ecc4a15be34f0d8370841f2, State: Initialized, Role: FOLLOWER
I20260619 06:44:59.138180 29077 consensus_queue.cc:260] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81ebb9d2ecc4a15be34f0d8370841f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } }
I20260619 06:44:59.138406 29077 raft_consensus.cc:388] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:44:59.138542 29077 raft_consensus.cc:482] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:44:59.138705 29077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:44:59.141178 29077 raft_consensus.cc:504] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81ebb9d2ecc4a15be34f0d8370841f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } }
I20260619 06:44:59.141732 29077 leader_election.cc:302] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f81ebb9d2ecc4a15be34f0d8370841f2; no voters:
I20260619 06:44:59.142225 29077 leader_election.cc:288] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:44:59.142359 29087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:44:59.142947 29087 raft_consensus.cc:686] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [term 1 LEADER]: Becoming Leader. State: Replica: f81ebb9d2ecc4a15be34f0d8370841f2, State: Running, Role: LEADER
I20260619 06:44:59.143440 29087 consensus_queue.cc:237] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81ebb9d2ecc4a15be34f0d8370841f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } }
I20260619 06:44:59.144147 29077 sys_catalog.cc:564] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:44:59.146476 29089 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f81ebb9d2ecc4a15be34f0d8370841f2. Latest consensus state: current_term: 1 leader_uuid: "f81ebb9d2ecc4a15be34f0d8370841f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81ebb9d2ecc4a15be34f0d8370841f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } } }
I20260619 06:44:59.146919 29089 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:59.146855 29088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f81ebb9d2ecc4a15be34f0d8370841f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f81ebb9d2ecc4a15be34f0d8370841f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37619 } } }
I20260619 06:44:59.147141 29088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2 [sys.catalog]: This master's current role is: LEADER
I20260619 06:44:59.147825 29094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:44:59.154510 29094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:44:59.164609 29094 catalog_manager.cc:1347] Generated new cluster ID: 4d72b9a26a454db187650c83507c8825
I20260619 06:44:59.164762 29094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:44:59.176669 29094 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:44:59.177331 29094 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:44:59.193228 29094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f81ebb9d2ecc4a15be34f0d8370841f2: Generated new TSK 0
I20260619 06:44:59.194180 29094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:44:59.206935 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29010
W20260619 06:44:59.208169 29072 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.156:60831
W20260619 06:44:59.208889 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29010: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:44:59.209049 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 29010
2026-06-19T06:44:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1775 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:44:59 dist-test-slave-qfk4 krb5kdc[29108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:44:59 dist-test-slave-qfk4 krb5kdc[29108](info): set up 2 sockets
Jun 19 06:44:59 dist-test-slave-qfk4 krb5kdc[29108](info): commencing operation
krb5kdc: starting...
W20260619 06:45:00.244957 7755 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:45:00 dist-test-slave-qfk4 krb5kdc[29108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:45:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:45:00.491577 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:45:00.815389 29121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:00.815670 29121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:00.815762 29121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:00.822960 29121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:00.823081 29121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:00.823138 29121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:00.823222 29121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:00.823282 29121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:00.833022 29121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:00.834336 29121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:00.835654 29121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:00.847170 29126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:00.847276 29127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:00.847462 29129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:00.848117 29121 server_base.cc:1034] running on GCE node
I20260619 06:45:00.849009 29121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:00.850474 29121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:00.851704 29121 hybrid_clock.cc:648] HybridClock initialized: now 1781851500851658 us; error 33 us; skew 500 ppm
I20260619 06:45:00.852113 29121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:00 dist-test-slave-qfk4 krb5kdc[29108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:00.855437 29121 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:45:00.857134 29121 webserver.cc:460] Webserver started at http://127.0.0.1:38063/ using document root <none> and password file <none>
I20260619 06:45:00.857843 29121 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:00.857968 29121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:00.858260 29121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:00.861393 29121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "f39f9385f90d47a5827fe1ca332223ae"
format_stamp: "Formatted at 2026-06-19 06:45:00 on dist-test-slave-qfk4"
server_key: "205b3ac5dd8c0d073c16a58836bb07ea"
server_key_iv: "2d9d7b503f476b08dce12b39fd814908"
server_key_version: "encryptionkey@0"
I20260619 06:45:00.862171 29121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "f39f9385f90d47a5827fe1ca332223ae"
format_stamp: "Formatted at 2026-06-19 06:45:00 on dist-test-slave-qfk4"
server_key: "205b3ac5dd8c0d073c16a58836bb07ea"
server_key_iv: "2d9d7b503f476b08dce12b39fd814908"
server_key_version: "encryptionkey@0"
I20260619 06:45:00.867571 29121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:45:00.871147 29136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:00.872421 29121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:45:00.872610 29121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "f39f9385f90d47a5827fe1ca332223ae"
format_stamp: "Formatted at 2026-06-19 06:45:00 on dist-test-slave-qfk4"
server_key: "205b3ac5dd8c0d073c16a58836bb07ea"
server_key_iv: "2d9d7b503f476b08dce12b39fd814908"
server_key_version: "encryptionkey@0"
I20260619 06:45:00.872792 29121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:00.885095 29121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:00.889139 29121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:00.889390 29121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:00.906569 29121 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37059
I20260619 06:45:00.906574 29187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37059 every 8 connection(s)
I20260619 06:45:00.908087 29121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:45:00.911844 29188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:00.915088 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 29121
I20260619 06:45:00.915264 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:45:00.915671 7755 external_mini_cluster.cc:1428] Setting key 0a7110eff7a6272d163c8fa21c912dc0
I20260619 06:45:00.924026 29188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae: Bootstrap starting.
I20260619 06:45:00.927891 29188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:00.929253 29188 log.cc:824] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:45:00 dist-test-slave-qfk4 krb5kdc[29108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:45:00.932514 29188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae: No bootstrap required, opened a new log
I20260619 06:45:00.938915 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:00.917481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:58795 (local address 127.0.0.1:37059)
0619 06:45:00.918184 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:00.918196 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:00.918238 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:45:00.919907 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:00.919931 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:00.920367 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:00.920770 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:00.922426 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:00.923766 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:00.925534 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:00.926233 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:00.926358 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:00.931557 (+ 5199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:00.931566 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:00.931581 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:00.931621 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:00.934634 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:00.935588 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:00.935594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:00.935601 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:00.935773 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:00.936233 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:00.936236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:00.936238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:00.936809 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:00.937060 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:00.937507 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 06:45:00.937836 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":307,"threads_started":1}
I20260619 06:45:00.941521 29188 raft_consensus.cc:348] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39f9385f90d47a5827fe1ca332223ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260619 06:45:00.941949 29188 raft_consensus.cc:374] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:45:00.942041 29188 raft_consensus.cc:729] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39f9385f90d47a5827fe1ca332223ae, State: Initialized, Role: FOLLOWER
I20260619 06:45:00.942633 29188 consensus_queue.cc:260] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39f9385f90d47a5827fe1ca332223ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260619 06:45:00.942889 29188 raft_consensus.cc:388] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:45:00.942982 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:45:00.943024 29188 raft_consensus.cc:482] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:45:00.943188 29188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:00.945348 29188 raft_consensus.cc:504] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39f9385f90d47a5827fe1ca332223ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260619 06:45:00.945876 29188 leader_election.cc:302] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f39f9385f90d47a5827fe1ca332223ae; no voters:
I20260619 06:45:00.946465 29188 leader_election.cc:288] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:45:00.946594 29197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:45:00.947161 29197 raft_consensus.cc:686] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [term 1 LEADER]: Becoming Leader. State: Replica: f39f9385f90d47a5827fe1ca332223ae, State: Running, Role: LEADER
I20260619 06:45:00.947703 29197 consensus_queue.cc:237] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39f9385f90d47a5827fe1ca332223ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260619 06:45:00.949831 29188 sys_catalog.cc:564] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:00.950719 29199 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f39f9385f90d47a5827fe1ca332223ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39f9385f90d47a5827fe1ca332223ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } } }
I20260619 06:45:00.950771 29198 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader f39f9385f90d47a5827fe1ca332223ae. Latest consensus state: current_term: 1 leader_uuid: "f39f9385f90d47a5827fe1ca332223ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39f9385f90d47a5827fe1ca332223ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } } }
I20260619 06:45:00.951145 29199 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:00.951140 29198 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:00.960193 29203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:45:00.967340 29203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:45:00.967762 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:00.947721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:38093 (local address 127.0.0.1:37059)
0619 06:45:00.947888 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:00.947893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:00.948208 (+ 315us) server_negotiation.cc:407] Connection header received
0619 06:45:00.948594 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:00.948599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:00.948681 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:00.948785 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:00.951754 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:00.953013 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:00.955053 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:00.955800 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:00.955944 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:00.959302 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:00.959310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:00.959316 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:00.959364 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:00.963419 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:00.964523 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:00.964541 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:00.964545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:00.964757 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:00.965351 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:00.965356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:00.965359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:00.965776 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:00.966099 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:00.967203 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0619 06:45:00.967393 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 06:45:00.978158 29203 catalog_manager.cc:1347] Generated new cluster ID: e40c878dbce64781aadce0d7f55d6880
I20260619 06:45:00.978278 29203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:45:00.992113 29203 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:45:00.993537 29203 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:45:00.999964 29203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f39f9385f90d47a5827fe1ca332223ae: Generated new TSK 0
I20260619 06:45:01.000869 29203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:45:01.008831 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29121
I20260619 06:45:01.010255 29183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:45:01.099067 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 29121
W20260619 06:45:01.112622 29106 connection.cc:535] client connection to 127.0.0.1:37059 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37059): Connection reset by peer (error 104)
2026-06-19T06:45:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1890 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:45:01 dist-test-slave-qfk4 krb5kdc[29220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:45:01 dist-test-slave-qfk4 krb5kdc[29220](info): set up 2 sockets
Jun 19 06:45:01 dist-test-slave-qfk4 krb5kdc[29220](info): commencing operation
krb5kdc: starting...
W20260619 06:45:02.116954 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:45:02 dist-test-slave-qfk4 krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:45:02.366459 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:45:02.694864 29233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:02.695164 29233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:02.695286 29233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:02.702559 29233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:02.702672 29233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:02.702740 29233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:02.702816 29233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:02.702873 29233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:02.712865 29233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:02.714150 29233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:02.715430 29233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:02.726958 29238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:02.726897 29241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:02.726866 29239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:02.727609 29233 server_base.cc:1034] running on GCE node
I20260619 06:45:02.728343 29233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:02.729907 29233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:02.731122 29233 hybrid_clock.cc:648] HybridClock initialized: now 1781851502731077 us; error 40 us; skew 500 ppm
I20260619 06:45:02.731456 29233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:02 dist-test-slave-qfk4 krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:02.737740 29233 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:45:02.739315 29233 webserver.cc:460] Webserver started at http://127.0.0.1:43581/ using document root <none> and password file <none>
I20260619 06:45:02.739964 29233 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:02.740085 29233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:02.740357 29233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:02.743302 29233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "5ff80238b3cc4f26bf98f92aa2410921"
format_stamp: "Formatted at 2026-06-19 06:45:02 on dist-test-slave-qfk4"
server_key: "79eeb66234cc13554d817df711a90547"
server_key_iv: "917d791579511f9e0f382542e60fdd0a"
server_key_version: "encryptionkey@0"
I20260619 06:45:02.744035 29233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "5ff80238b3cc4f26bf98f92aa2410921"
format_stamp: "Formatted at 2026-06-19 06:45:02 on dist-test-slave-qfk4"
server_key: "79eeb66234cc13554d817df711a90547"
server_key_iv: "917d791579511f9e0f382542e60fdd0a"
server_key_version: "encryptionkey@0"
I20260619 06:45:02.749022 29233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:45:02.752519 29248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:02.753970 29233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:45:02.754181 29233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "5ff80238b3cc4f26bf98f92aa2410921"
format_stamp: "Formatted at 2026-06-19 06:45:02 on dist-test-slave-qfk4"
server_key: "79eeb66234cc13554d817df711a90547"
server_key_iv: "917d791579511f9e0f382542e60fdd0a"
server_key_version: "encryptionkey@0"
I20260619 06:45:02.754366 29233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:02.769125 29233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:02.773440 29233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:02.773701 29233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:02.791230 29233 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36243
I20260619 06:45:02.791258 29299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36243 every 8 connection(s)
I20260619 06:45:02.792614 29233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:45:02.794934 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 29233
I20260619 06:45:02.795104 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:45:02.795670 7755 external_mini_cluster.cc:1428] Setting key 53c49c481ee6397f67ab57dd3b832f6d
I20260619 06:45:02.797029 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:02.808354 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:02.797615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:55507 (local address 127.0.0.1:36243)
0619 06:45:02.798355 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:02.798367 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:02.798407 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:45:02.799997 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:02.800019 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:02.800439 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:02.800878 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:02.801408 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:02.802735 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:02.804050 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:02.804785 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:02.804945 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:02.806121 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:02.806129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:02.806146 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:02.806717 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:02.806796 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:02.806831 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:02.806970 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:02.807248 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 06:45:02.807409 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":336,"threads_started":1}
I20260619 06:45:02.808898 29300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921: Bootstrap starting.
I20260619 06:45:02.812024 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:45:02.812068 29300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:02.813108 29300 log.cc:824] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:02.815850 29300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921: No bootstrap required, opened a new log
I20260619 06:45:02.822077 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:02.816211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:43397 (local address 127.0.0.1:36243)
0619 06:45:02.816365 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:02.816372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:02.816797 (+ 425us) server_negotiation.cc:407] Connection header received
0619 06:45:02.816974 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:02.816978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:02.817036 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:02.817123 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:02.817630 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:02.818409 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:02.819688 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:02.820181 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:02.820291 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:02.821359 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:02.821364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:02.821367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:02.821384 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:02.821459 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:02.821476 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:02.821571 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:02.821773 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 06:45:02.821831 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:45:02.823791 29300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff80238b3cc4f26bf98f92aa2410921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36243 } }
I20260619 06:45:02.824141 29300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:45:02.824230 29300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff80238b3cc4f26bf98f92aa2410921, State: Initialized, Role: FOLLOWER
I20260619 06:45:02.824811 29300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff80238b3cc4f26bf98f92aa2410921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36243 } }
I20260619 06:45:02.825012 29300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:45:02.825138 29300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:45:02.825289 29300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:02.827450 29300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff80238b3cc4f26bf98f92aa2410921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36243 } }
I20260619 06:45:02.827948 29300 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ff80238b3cc4f26bf98f92aa2410921; no voters:
I20260619 06:45:02.828459 29300 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:45:02.828595 29310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:45:02.829336 29310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff80238b3cc4f26bf98f92aa2410921, State: Running, Role: LEADER
I20260619 06:45:02.829823 29310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff80238b3cc4f26bf98f92aa2410921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36243 } }
I20260619 06:45:02.830592 29300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:02.832538 29312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ff80238b3cc4f26bf98f92aa2410921. Latest consensus state: current_term: 1 leader_uuid: "5ff80238b3cc4f26bf98f92aa2410921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff80238b3cc4f26bf98f92aa2410921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36243 } } }
I20260619 06:45:02.832976 29312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:02.832537 29311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ff80238b3cc4f26bf98f92aa2410921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff80238b3cc4f26bf98f92aa2410921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36243 } } }
I20260619 06:45:02.833428 29311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:02.837149 29318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:45:02.842638 29318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:45:02.850834 29318 catalog_manager.cc:1347] Generated new cluster ID: d09d576d38314b60a8e27670c72edba8
I20260619 06:45:02.850950 29318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:45:02.892331 29318 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:45:02.894256 29318 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:45:02.903090 29318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ff80238b3cc4f26bf98f92aa2410921: Generated new TSK 0
I20260619 06:45:02.904183 29318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:45:02.910333 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29233
W20260619 06:45:02.911715 29295 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.156:55507
W20260619 06:45:02.912528 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29233: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:45:02.912706 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 29233
W20260619 06:45:02.926064 29218 connection.cc:535] client connection to 127.0.0.1:36243 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36243): Connection reset by peer (error 104)
2026-06-19T06:45:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1813 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:45:03 dist-test-slave-qfk4 krb5kdc[29331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:45:03 dist-test-slave-qfk4 krb5kdc[29331](info): set up 2 sockets
Jun 19 06:45:03 dist-test-slave-qfk4 krb5kdc[29331](info): commencing operation
krb5kdc: starting...
W20260619 06:45:03.949055 7755 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:45:04 dist-test-slave-qfk4 krb5kdc[29331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:45:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:45:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:45:04.195483 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:45:04.514420 29344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:04.514709 29344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:04.514829 29344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:04.522056 29344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:04.522182 29344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:04.522239 29344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:04.522325 29344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:04.522384 29344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:04.532358 29344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:04.533767 29344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:04.535117 29344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:04.547221 29352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:04.547178 29350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:04.547175 29349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:04.548518 29344 server_base.cc:1034] running on GCE node
I20260619 06:45:04.549350 29344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:04.550858 29344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:04.552081 29344 hybrid_clock.cc:648] HybridClock initialized: now 1781851504552023 us; error 57 us; skew 500 ppm
I20260619 06:45:04.552453 29344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:04 dist-test-slave-qfk4 krb5kdc[29331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:04.555641 29344 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:45:04.557236 29344 webserver.cc:460] Webserver started at http://127.0.0.1:42709/ using document root <none> and password file <none>
I20260619 06:45:04.557878 29344 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:04.557974 29344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:04.558249 29344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:04.561295 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "8d23ff4ec8254936a71b7d1175b15385"
format_stamp: "Formatted at 2026-06-19 06:45:04 on dist-test-slave-qfk4"
server_key: "fe608e1766c5067f0a4646339eb5a2c1"
server_key_iv: "f8ac566242ef0c64ba28994999503e25"
server_key_version: "encryptionkey@0"
I20260619 06:45:04.562047 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "8d23ff4ec8254936a71b7d1175b15385"
format_stamp: "Formatted at 2026-06-19 06:45:04 on dist-test-slave-qfk4"
server_key: "fe608e1766c5067f0a4646339eb5a2c1"
server_key_iv: "f8ac566242ef0c64ba28994999503e25"
server_key_version: "encryptionkey@0"
I20260619 06:45:04.567400 29344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:45:04.571161 29359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:04.572700 29344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 06:45:04.572881 29344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "8d23ff4ec8254936a71b7d1175b15385"
format_stamp: "Formatted at 2026-06-19 06:45:04 on dist-test-slave-qfk4"
server_key: "fe608e1766c5067f0a4646339eb5a2c1"
server_key_iv: "f8ac566242ef0c64ba28994999503e25"
server_key_version: "encryptionkey@0"
I20260619 06:45:04.573138 29344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:04.578979 29344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:04.583288 29344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:04.583578 29344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:04.601775 29344 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44525
I20260619 06:45:04.601802 29410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44525 every 8 connection(s)
I20260619 06:45:04.603303 29344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:45:04.607240 29411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:04.608343 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 29344
I20260619 06:45:04.608520 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:45:04.609150 7755 external_mini_cluster.cc:1428] Setting key d44aa43d4cef2c55206c6c19b49f88eb
I20260619 06:45:04.617250 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:04.611056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:37391 (local address 127.0.0.1:44525)
0619 06:45:04.611833 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:04.611845 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:04.611887 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:45:04.613819 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:04.613846 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:04.614203 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:04.614589 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:04.615614 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:04.615620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:04.615633 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:04.615648 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:04.615708 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:04.615731 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:04.615929 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:04.616067 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 06:45:04.616260 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":341,"threads_started":1}
I20260619 06:45:04.620668 29411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385: Bootstrap starting.
I20260619 06:45:04.621172 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:45:04.623929 29411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:04.625046 29411 log.cc:824] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:04.631101 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:04.625522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:36423 (local address 127.0.0.1:44525)
0619 06:45:04.625669 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:04.625674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:04.629162 (+ 3488us) server_negotiation.cc:407] Connection header received
0619 06:45:04.629339 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:04.629344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:04.629398 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:04.629501 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:04.630449 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:04.630454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:04.630457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:04.630472 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:04.630521 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:04.630538 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:04.630638 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:04.630757 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 06:45:04.630821 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:45:04.631717 29411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385: No bootstrap required, opened a new log
I20260619 06:45:04.639415 29411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d23ff4ec8254936a71b7d1175b15385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } }
I20260619 06:45:04.639819 29411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:45:04.639914 29411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d23ff4ec8254936a71b7d1175b15385, State: Initialized, Role: FOLLOWER
I20260619 06:45:04.640630 29411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d23ff4ec8254936a71b7d1175b15385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } }
I20260619 06:45:04.640909 29411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:45:04.641062 29411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:45:04.641248 29411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:04.643501 29411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d23ff4ec8254936a71b7d1175b15385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } }
I20260619 06:45:04.643997 29411 leader_election.cc:302] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d23ff4ec8254936a71b7d1175b15385; no voters:
I20260619 06:45:04.644554 29411 leader_election.cc:288] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:45:04.644740 29421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:45:04.645438 29421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [term 1 LEADER]: Becoming Leader. State: Replica: 8d23ff4ec8254936a71b7d1175b15385, State: Running, Role: LEADER
I20260619 06:45:04.645960 29421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d23ff4ec8254936a71b7d1175b15385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } }
I20260619 06:45:04.646672 29411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:04.649120 29422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d23ff4ec8254936a71b7d1175b15385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d23ff4ec8254936a71b7d1175b15385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } } }
I20260619 06:45:04.649072 29423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d23ff4ec8254936a71b7d1175b15385. Latest consensus state: current_term: 1 leader_uuid: "8d23ff4ec8254936a71b7d1175b15385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d23ff4ec8254936a71b7d1175b15385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44525 } } }
I20260619 06:45:04.649710 29422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:04.649768 29423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:04.650676 29429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:45:04.656667 29429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:45:04.666945 29429 catalog_manager.cc:1347] Generated new cluster ID: 8bc4266ef068448b9818d771bd081d97
I20260619 06:45:04.667085 29429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:45:04.686254 29429 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:45:04.686721 29429 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:45:04.699303 29429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d23ff4ec8254936a71b7d1175b15385: Generated new TSK 0
I20260619 06:45:04.700268 29429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:45:04.712841 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29344
W20260619 06:45:04.713968 29406 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.156:37391
W20260619 06:45:04.714668 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29344: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:45:04.714797 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 29344
2026-06-19T06:45:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1804 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:45:04 dist-test-slave-qfk4 krb5kdc[29442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:45:04 dist-test-slave-qfk4 krb5kdc[29442](info): set up 2 sockets
Jun 19 06:45:04 dist-test-slave-qfk4 krb5kdc[29442](info): commencing operation
krb5kdc: starting...
W20260619 06:45:05.748929 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:45:05 dist-test-slave-qfk4 krb5kdc[29442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:45:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:45:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:45:05.987278 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:32981
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:35485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:45:06.313135 29455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:06.313406 29455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:06.313541 29455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:06.320696 29455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:06.320806 29455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:06.320860 29455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:06.320943 29455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:06.321004 29455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:06.330796 29455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:35485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:32981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:06.332062 29455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:06.333359 29455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:06.345247 29460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:06.345203 29461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:06.345290 29463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:06.346329 29455 server_base.cc:1034] running on GCE node
I20260619 06:45:06.347198 29455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:06.348752 29455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:06.349982 29455 hybrid_clock.cc:648] HybridClock initialized: now 1781851506349935 us; error 32 us; skew 500 ppm
I20260619 06:45:06.350358 29455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:06 dist-test-slave-qfk4 krb5kdc[29442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851506, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:06.353554 29455 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:45:06.354995 29455 webserver.cc:460] Webserver started at http://127.7.146.254:43733/ using document root <none> and password file <none>
I20260619 06:45:06.355675 29455 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:06.355780 29455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:06.356067 29455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:06.359033 29455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "39b2a09c5e3744d1b120b98abcd2b764"
format_stamp: "Formatted at 2026-06-19 06:45:06 on dist-test-slave-qfk4"
server_key: "d09162154d71c0437f049708bd337ec4"
server_key_iv: "51c880be5185cf86ac7afedd26b14df2"
server_key_version: "encryptionkey@0"
I20260619 06:45:06.359872 29455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "39b2a09c5e3744d1b120b98abcd2b764"
format_stamp: "Formatted at 2026-06-19 06:45:06 on dist-test-slave-qfk4"
server_key: "d09162154d71c0437f049708bd337ec4"
server_key_iv: "51c880be5185cf86ac7afedd26b14df2"
server_key_version: "encryptionkey@0"
I20260619 06:45:06.364989 29455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 06:45:06.368381 29470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:06.369719 29455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:45:06.369871 29455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "39b2a09c5e3744d1b120b98abcd2b764"
format_stamp: "Formatted at 2026-06-19 06:45:06 on dist-test-slave-qfk4"
server_key: "d09162154d71c0437f049708bd337ec4"
server_key_iv: "51c880be5185cf86ac7afedd26b14df2"
server_key_version: "encryptionkey@0"
I20260619 06:45:06.370044 29455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:06.395800 29455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:06.400159 29455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:06.400424 29455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:06.417743 29455 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:32981
I20260619 06:45:06.417763 29521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:32981 every 8 connection(s)
I20260619 06:45:06.419064 29455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:45:06.420845 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 29455
I20260619 06:45:06.421021 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:45:06.421510 7755 external_mini_cluster.cc:1428] Setting key fabb483f675bea69552ebd22971954ee
I20260619 06:45:06.423278 29522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:06.434252 29522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764: Bootstrap starting.
Jun 19 06:45:06 dist-test-slave-qfk4 krb5kdc[29442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:45:06.437747 29522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:06.438745 29522 log.cc:824] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:06.441852 29522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764: No bootstrap required, opened a new log
I20260619 06:45:06.443455 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:06.423443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51387 (local address 127.7.146.254:32981)
0619 06:45:06.424192 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:06.424204 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:06.424243 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:45:06.425898 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:06.425921 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:06.426336 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:06.426729 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:06.428092 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:06.429533 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:06.430842 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:06.431528 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:06.431694 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:06.436039 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:06.436047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:06.436062 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:06.436097 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:06.439031 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:06.439827 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:06.439834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:06.439842 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:06.440011 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:06.440534 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:06.440537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:06.440539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:06.441132 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:06.441361 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:06.441959 (+ 598us) server_negotiation.cc:299] Negotiation successful
0619 06:45:06.442360 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":318,"threads_started":1}
I20260619 06:45:06.447505 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:45:06.449906 29522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2a09c5e3744d1b120b98abcd2b764" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 32981 } }
I20260619 06:45:06.450387 29522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:45:06.450511 29522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b2a09c5e3744d1b120b98abcd2b764, State: Initialized, Role: FOLLOWER
I20260619 06:45:06.451156 29522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2a09c5e3744d1b120b98abcd2b764" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 32981 } }
I20260619 06:45:06.451361 29522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:45:06.451503 29522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:45:06.451704 29522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:06.454136 29522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2a09c5e3744d1b120b98abcd2b764" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 32981 } }
I20260619 06:45:06.454697 29522 leader_election.cc:302] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39b2a09c5e3744d1b120b98abcd2b764; no voters:
I20260619 06:45:06.455371 29522 leader_election.cc:288] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:45:06.455473 29532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:45:06.456846 29532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [term 1 LEADER]: Becoming Leader. State: Replica: 39b2a09c5e3744d1b120b98abcd2b764, State: Running, Role: LEADER
I20260619 06:45:06.457551 29522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:06.457479 29532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2a09c5e3744d1b120b98abcd2b764" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 32981 } }
I20260619 06:45:06.461802 29534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39b2a09c5e3744d1b120b98abcd2b764. Latest consensus state: current_term: 1 leader_uuid: "39b2a09c5e3744d1b120b98abcd2b764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2a09c5e3744d1b120b98abcd2b764" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 32981 } } }
I20260619 06:45:06.462133 29534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:06.462538 29533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39b2a09c5e3744d1b120b98abcd2b764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2a09c5e3744d1b120b98abcd2b764" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 32981 } } }
I20260619 06:45:06.462877 29533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:06.463224 29540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:45:06.469648 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:06.451834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50951 (local address 127.7.146.254:32981)
0619 06:45:06.451991 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:06.451996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:06.452472 (+ 476us) server_negotiation.cc:407] Connection header received
0619 06:45:06.452688 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:06.452693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:06.452767 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:06.452855 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:06.454377 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:06.455533 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:06.457392 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:06.458039 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:06.458171 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:06.461563 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:06.461571 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:06.461576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:06.461622 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:06.465706 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:06.466826 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:06.466833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:06.466837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:06.467010 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:06.467576 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:06.467582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:06.467586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:06.467983 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:06.468284 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:06.468997 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 06:45:06.469211 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 06:45:06.476663 29540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:45:06.486280 29540 catalog_manager.cc:1347] Generated new cluster ID: 6b6a4c2bac774d849a6c14f4efeab37d
I20260619 06:45:06.486399 29540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:45:06.505290 29540 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:45:06.507076 29540 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:45:06.519965 29540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39b2a09c5e3744d1b120b98abcd2b764: Generated new TSK 0
I20260619 06:45:06.521211 29540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:45:06.532764 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29455
I20260619 06:45:06.534336 29517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:45:06.626119 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 29455
W20260619 06:45:06.639451 29440 connection.cc:535] client connection to 127.7.146.254:32981 recv error: Network error: failed to read from TLS socket (remote: 127.7.146.254:32981): Connection reset by peer (error 104)
2026-06-19T06:45:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:45:06 dist-test-slave-qfk4 krb5kdc[29554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:45:06 dist-test-slave-qfk4 krb5kdc[29554](info): set up 2 sockets
Jun 19 06:45:06 dist-test-slave-qfk4 krb5kdc[29554](info): commencing operation
krb5kdc: starting...
W20260619 06:45:07.644971 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:45:07 dist-test-slave-qfk4 krb5kdc[29554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:45:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:45:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:45:07.894320 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40751
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:40751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:45:08.224520 29567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:08.224905 29567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:08.225057 29567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:08.232396 29567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:08.232539 29567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:08.232640 29567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:08.232719 29567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:08.232779 29567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:08.242686 29567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:40751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:40751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:08.243975 29567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:08.245287 29567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:08.256608 29572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:08.256557 29573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:08.256971 29575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:08.257454 29567 server_base.cc:1034] running on GCE node
I20260619 06:45:08.258224 29567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:08.259644 29567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:08.260841 29567 hybrid_clock.cc:648] HybridClock initialized: now 1781851508260794 us; error 46 us; skew 500 ppm
I20260619 06:45:08.261190 29567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:08 dist-test-slave-qfk4 krb5kdc[29554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851508, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:08.268517 29567 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:45:08.270212 29567 webserver.cc:460] Webserver started at http://127.7.146.254:34631/ using document root <none> and password file <none>
I20260619 06:45:08.270874 29567 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:08.270991 29567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:08.271353 29567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:08.274426 29567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "9fbf9c43c7954d6f87bba2a31b2db98c"
format_stamp: "Formatted at 2026-06-19 06:45:08 on dist-test-slave-qfk4"
server_key: "5464d835e7b140f760e36725f96a0705"
server_key_iv: "fb97cb2b15d4c36275f5e41ada853750"
server_key_version: "encryptionkey@0"
I20260619 06:45:08.275187 29567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "9fbf9c43c7954d6f87bba2a31b2db98c"
format_stamp: "Formatted at 2026-06-19 06:45:08 on dist-test-slave-qfk4"
server_key: "5464d835e7b140f760e36725f96a0705"
server_key_iv: "fb97cb2b15d4c36275f5e41ada853750"
server_key_version: "encryptionkey@0"
I20260619 06:45:08.280552 29567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 06:45:08.284229 29582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:08.285655 29567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:45:08.285820 29567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "9fbf9c43c7954d6f87bba2a31b2db98c"
format_stamp: "Formatted at 2026-06-19 06:45:08 on dist-test-slave-qfk4"
server_key: "5464d835e7b140f760e36725f96a0705"
server_key_iv: "fb97cb2b15d4c36275f5e41ada853750"
server_key_version: "encryptionkey@0"
I20260619 06:45:08.286008 29567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:08.297513 29567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:08.301532 29567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:08.301791 29567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:08.319295 29567 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:40751
I20260619 06:45:08.319315 29633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:40751 every 8 connection(s)
I20260619 06:45:08.320715 29567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:45:08.324448 29634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:08.328655 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 29567
I20260619 06:45:08.328830 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:45:08.329360 7755 external_mini_cluster.cc:1428] Setting key 7e4ef21fcd9b6add4ac94d0fd3402d2f
I20260619 06:45:08.337280 29634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c: Bootstrap starting.
I20260619 06:45:08.340428 29634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:08.341588 29634 log.cc:824] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:08.342216 29637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:08.331254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53501 (local address 127.7.146.254:40751)
0619 06:45:08.332027 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:08.332040 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:08.332079 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:45:08.333853 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:08.333879 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:08.334319 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:08.334698 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:08.335262 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:08.336542 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:08.337864 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:08.338528 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:08.338706 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:08.339907 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:08.339914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:08.339929 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:08.340483 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:08.340596 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:08.340632 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:08.340791 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:08.341042 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 06:45:08.341199 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":350,"threads_started":1}
I20260619 06:45:08.344419 29634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c: No bootstrap required, opened a new log
I20260619 06:45:08.346108 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:45:08.353993 29634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbf9c43c7954d6f87bba2a31b2db98c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40751 } }
I20260619 06:45:08.354423 29634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:45:08.354526 29634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbf9c43c7954d6f87bba2a31b2db98c, State: Initialized, Role: FOLLOWER
I20260619 06:45:08.355190 29634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbf9c43c7954d6f87bba2a31b2db98c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40751 } }
I20260619 06:45:08.355468 29634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:45:08.355604 29634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:45:08.355768 29634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:08.358667 29634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbf9c43c7954d6f87bba2a31b2db98c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40751 } }
I20260619 06:45:08.359302 29637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:08.351145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45511 (local address 127.7.146.254:40751)
0619 06:45:08.351345 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:08.351351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:08.353116 (+ 1765us) server_negotiation.cc:407] Connection header received
0619 06:45:08.353293 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:08.353298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:08.353363 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:08.353456 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:08.354008 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:08.354858 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:08.356663 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:08.357201 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:08.357298 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:08.358472 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:08.358478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:08.358482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:08.358501 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:08.358550 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:08.358569 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:08.358672 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:08.358921 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 06:45:08.358980 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:45:08.359330 29634 leader_election.cc:302] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fbf9c43c7954d6f87bba2a31b2db98c; no voters:
I20260619 06:45:08.359982 29634 leader_election.cc:288] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:45:08.360136 29644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:45:08.360848 29644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [term 1 LEADER]: Becoming Leader. State: Replica: 9fbf9c43c7954d6f87bba2a31b2db98c, State: Running, Role: LEADER
I20260619 06:45:08.361766 29644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbf9c43c7954d6f87bba2a31b2db98c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40751 } }
I20260619 06:45:08.362715 29634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:08.364428 29646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fbf9c43c7954d6f87bba2a31b2db98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbf9c43c7954d6f87bba2a31b2db98c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40751 } } }
I20260619 06:45:08.365223 29646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:08.364709 29645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fbf9c43c7954d6f87bba2a31b2db98c. Latest consensus state: current_term: 1 leader_uuid: "9fbf9c43c7954d6f87bba2a31b2db98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbf9c43c7954d6f87bba2a31b2db98c" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 40751 } } }
I20260619 06:45:08.365734 29645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:08.373027 29650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:45:08.378294 29650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:45:08.387432 29650 catalog_manager.cc:1347] Generated new cluster ID: e4652d09dfde4b2f8ff54115a4ec7295
I20260619 06:45:08.387545 29650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:45:08.413754 29650 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:45:08.415071 29650 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:45:08.421896 29650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fbf9c43c7954d6f87bba2a31b2db98c: Generated new TSK 0
I20260619 06:45:08.422923 29650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:45:08.436455 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29567
W20260619 06:45:08.437728 29629 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53501
W20260619 06:45:08.438418 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29567: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:45:08.438576 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 29567
W20260619 06:45:08.452078 29552 connection.cc:535] client connection to 127.7.146.254:40751 recv error: Network error: failed to read from TLS socket (remote: 127.7.146.254:40751): Connection reset by peer (error 104)
2026-06-19T06:45:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1812 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:45:08 dist-test-slave-qfk4 krb5kdc[29665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:45:08 dist-test-slave-qfk4 krb5kdc[29665](info): set up 2 sockets
Jun 19 06:45:08 dist-test-slave-qfk4 krb5kdc[29665](info): commencing operation
krb5kdc: starting...
W20260619 06:45:09.448809 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:45:09 dist-test-slave-qfk4 krb5kdc[29665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:45:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:45:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:45:09.688416 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:45735
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:45735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:45:10.020910 29678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:10.021234 29678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:10.021369 29678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:10.028789 29678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:10.028920 29678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:10.028997 29678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:10.029073 29678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:10.029132 29678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:10.039147 29678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:45735
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:45735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:10.040467 29678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:10.041826 29678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:10.053449 29683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:10.053503 29684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:10.053516 29686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:10.054342 29678 server_base.cc:1034] running on GCE node
I20260619 06:45:10.055186 29678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:10.056707 29678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:10.057928 29678 hybrid_clock.cc:648] HybridClock initialized: now 1781851510057880 us; error 44 us; skew 500 ppm
I20260619 06:45:10.058295 29678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:10 dist-test-slave-qfk4 krb5kdc[29665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851510, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:10.061431 29678 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:45:10.063016 29678 webserver.cc:460] Webserver started at http://127.7.146.254:36997/ using document root <none> and password file <none>
I20260619 06:45:10.063690 29678 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:10.063791 29678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:10.064042 29678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:10.067157 29678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "3b46e2b51dfc4abaa00072852731defd"
format_stamp: "Formatted at 2026-06-19 06:45:10 on dist-test-slave-qfk4"
server_key: "11c91e505dd8a09ff8666fdd4cb3ceba"
server_key_iv: "99c352f2fe1493b2b8539d22243e28a8"
server_key_version: "encryptionkey@0"
I20260619 06:45:10.067936 29678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "3b46e2b51dfc4abaa00072852731defd"
format_stamp: "Formatted at 2026-06-19 06:45:10 on dist-test-slave-qfk4"
server_key: "11c91e505dd8a09ff8666fdd4cb3ceba"
server_key_iv: "99c352f2fe1493b2b8539d22243e28a8"
server_key_version: "encryptionkey@0"
I20260619 06:45:10.073304 29678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 06:45:10.076932 29693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:10.078330 29678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:45:10.078490 29678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "3b46e2b51dfc4abaa00072852731defd"
format_stamp: "Formatted at 2026-06-19 06:45:10 on dist-test-slave-qfk4"
server_key: "11c91e505dd8a09ff8666fdd4cb3ceba"
server_key_iv: "99c352f2fe1493b2b8539d22243e28a8"
server_key_version: "encryptionkey@0"
I20260619 06:45:10.078685 29678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:10.084414 29678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:10.088685 29678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:10.088963 29678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:10.106467 29678 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:45735
I20260619 06:45:10.106504 29744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:45735 every 8 connection(s)
I20260619 06:45:10.107889 29678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:45:10.111565 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 29678
I20260619 06:45:10.111768 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:45:10.112075 29745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:10.112309 7755 external_mini_cluster.cc:1428] Setting key 3be3347a77f28ab5d24c45f76699e490
I20260619 06:45:10.120301 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:10.114162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45865 (local address 127.7.146.254:45735)
0619 06:45:10.114981 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:10.114992 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:10.115036 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:45:10.116921 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:10.116946 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:10.117322 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:10.117689 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:10.118748 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:10.118754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:10.118767 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:10.118781 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:10.118849 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:10.118868 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:10.119033 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:10.119138 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 06:45:10.119332 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":367,"threads_started":1}
I20260619 06:45:10.123937 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:45:10.125842 29745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd: Bootstrap starting.
I20260619 06:45:10.129730 29745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:10.130988 29745 log.cc:824] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:10.131367 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:10.129061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58433 (local address 127.7.146.254:45735)
0619 06:45:10.129236 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:10.129242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:10.129525 (+ 283us) server_negotiation.cc:407] Connection header received
0619 06:45:10.129694 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:10.129699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:10.129753 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:10.129839 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:10.130760 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:10.130764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:10.130767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:10.130781 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:10.130822 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:10.130836 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:10.130915 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:10.131053 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 06:45:10.131097 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:45:10.134028 29745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd: No bootstrap required, opened a new log
I20260619 06:45:10.141515 29745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b46e2b51dfc4abaa00072852731defd" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45735 } }
I20260619 06:45:10.141896 29745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:45:10.141983 29745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b46e2b51dfc4abaa00072852731defd, State: Initialized, Role: FOLLOWER
I20260619 06:45:10.142582 29745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b46e2b51dfc4abaa00072852731defd" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45735 } }
I20260619 06:45:10.142858 29745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:45:10.143016 29745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:45:10.143198 29745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:10.145388 29745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b46e2b51dfc4abaa00072852731defd" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45735 } }
I20260619 06:45:10.145836 29745 leader_election.cc:302] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b46e2b51dfc4abaa00072852731defd; no voters:
I20260619 06:45:10.146420 29745 leader_election.cc:288] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:45:10.146569 29755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:45:10.147296 29755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [term 1 LEADER]: Becoming Leader. State: Replica: 3b46e2b51dfc4abaa00072852731defd, State: Running, Role: LEADER
I20260619 06:45:10.147881 29755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b46e2b51dfc4abaa00072852731defd" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45735 } }
I20260619 06:45:10.148700 29745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:10.150872 29756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b46e2b51dfc4abaa00072852731defd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b46e2b51dfc4abaa00072852731defd" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45735 } } }
I20260619 06:45:10.151286 29756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:10.152406 29757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b46e2b51dfc4abaa00072852731defd. Latest consensus state: current_term: 1 leader_uuid: "3b46e2b51dfc4abaa00072852731defd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b46e2b51dfc4abaa00072852731defd" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 45735 } } }
I20260619 06:45:10.152731 29757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:10.153978 29762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:45:10.160048 29762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:45:10.171862 29762 catalog_manager.cc:1347] Generated new cluster ID: a3ac3449fa704559b29627f7aabb01a6
I20260619 06:45:10.171959 29762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:45:10.199429 29762 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:45:10.199966 29762 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:45:10.211544 29762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b46e2b51dfc4abaa00072852731defd: Generated new TSK 0
I20260619 06:45:10.212744 29762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:45:10.228045 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29678
W20260619 06:45:10.229430 29740 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45865
W20260619 06:45:10.230214 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29678: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:45:10.230350 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 29678
2026-06-19T06:45:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1791 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:45:10 dist-test-slave-qfk4 krb5kdc[29776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:45:10 dist-test-slave-qfk4 krb5kdc[29776](info): set up 2 sockets
Jun 19 06:45:10 dist-test-slave-qfk4 krb5kdc[29776](info): commencing operation
krb5kdc: starting...
W20260619 06:45:11.240940 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:45:11 dist-test-slave-qfk4 krb5kdc[29776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:45:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:45:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:45:11.477945 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:35581
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:35581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:45:11.794855 29789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:11.795171 29789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:11.795302 29789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:11.802569 29789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:11.802706 29789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:11.802788 29789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:11.802872 29789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:11.802932 29789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:11.812711 29789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:35581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:35581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:11.813967 29789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:11.815229 29789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:11.826457 29795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:11.826481 29794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:11.826647 29797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:11.827282 29789 server_base.cc:1034] running on GCE node
I20260619 06:45:11.828094 29789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:11.829635 29789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:11.830857 29789 hybrid_clock.cc:648] HybridClock initialized: now 1781851511830799 us; error 40 us; skew 500 ppm
I20260619 06:45:11.831194 29789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:11 dist-test-slave-qfk4 krb5kdc[29776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851511, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:11.834233 29789 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:45:11.835666 29789 webserver.cc:460] Webserver started at http://127.7.146.254:40147/ using document root <none> and password file <none>
I20260619 06:45:11.836320 29789 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:11.836433 29789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:11.836755 29789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:11.839758 29789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "1d4fd49304234bf980cb55d10e5e2416"
format_stamp: "Formatted at 2026-06-19 06:45:11 on dist-test-slave-qfk4"
server_key: "4ebcd07226274fc5e8acb4a8a6c8a0da"
server_key_iv: "26613599fdfee58c85c0036fffb045f9"
server_key_version: "encryptionkey@0"
I20260619 06:45:11.840536 29789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "1d4fd49304234bf980cb55d10e5e2416"
format_stamp: "Formatted at 2026-06-19 06:45:11 on dist-test-slave-qfk4"
server_key: "4ebcd07226274fc5e8acb4a8a6c8a0da"
server_key_iv: "26613599fdfee58c85c0036fffb045f9"
server_key_version: "encryptionkey@0"
I20260619 06:45:11.845685 29789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:45:11.849139 29804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:11.850605 29789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:45:11.850790 29789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "1d4fd49304234bf980cb55d10e5e2416"
format_stamp: "Formatted at 2026-06-19 06:45:11 on dist-test-slave-qfk4"
server_key: "4ebcd07226274fc5e8acb4a8a6c8a0da"
server_key_iv: "26613599fdfee58c85c0036fffb045f9"
server_key_version: "encryptionkey@0"
I20260619 06:45:11.851009 29789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:11.866923 29789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:11.871196 29789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:11.871496 29789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:11.888893 29789 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:35581
I20260619 06:45:11.888896 29855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:35581 every 8 connection(s)
I20260619 06:45:11.890452 29789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:45:11.890739 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 29789
I20260619 06:45:11.890893 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:45:11.891343 7755 external_mini_cluster.cc:1428] Setting key 6496fa580c0d65efc2869e828ce28af0
I20260619 06:45:11.895094 29856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:45:11 dist-test-slave-qfk4 krb5kdc[29776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:45:11.906678 29856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416: Bootstrap starting.
I20260619 06:45:11.910252 29856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:11.911373 29856 log.cc:824] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:11.913125 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:11.893369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41329 (local address 127.7.146.254:35581)
0619 06:45:11.894131 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:11.894143 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:11.894183 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:45:11.895821 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:11.895841 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:11.896235 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:11.896593 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:11.897994 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:11.899666 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:11.900950 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:11.901630 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:11.901790 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:11.905656 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:11.905664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:11.905680 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:11.905723 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:11.908853 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:11.909535 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:11.909542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:11.909549 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:11.909712 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:11.910083 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:11.910087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:11.910089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:11.910651 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:11.910932 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:11.911383 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 06:45:11.911786 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":345,"threads_started":1}
I20260619 06:45:11.914505 29856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416: No bootstrap required, opened a new log
I20260619 06:45:11.916952 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:45:11.922487 29856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4fd49304234bf980cb55d10e5e2416" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35581 } }
I20260619 06:45:11.922851 29856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:45:11.922945 29856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4fd49304234bf980cb55d10e5e2416, State: Initialized, Role: FOLLOWER
I20260619 06:45:11.923552 29856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4fd49304234bf980cb55d10e5e2416" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35581 } }
I20260619 06:45:11.923792 29856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:45:11.923919 29856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:45:11.924089 29856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:11.926347 29856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4fd49304234bf980cb55d10e5e2416" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35581 } }
I20260619 06:45:11.926880 29856 leader_election.cc:302] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d4fd49304234bf980cb55d10e5e2416; no voters:
I20260619 06:45:11.927434 29856 leader_election.cc:288] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:45:11.927601 29866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:45:11.928874 29866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [term 1 LEADER]: Becoming Leader. State: Replica: 1d4fd49304234bf980cb55d10e5e2416, State: Running, Role: LEADER
I20260619 06:45:11.929499 29866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4fd49304234bf980cb55d10e5e2416" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35581 } }
I20260619 06:45:11.929718 29856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:11.932106 29868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d4fd49304234bf980cb55d10e5e2416. Latest consensus state: current_term: 1 leader_uuid: "1d4fd49304234bf980cb55d10e5e2416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4fd49304234bf980cb55d10e5e2416" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35581 } } }
I20260619 06:45:11.932454 29868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:11.932993 29867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d4fd49304234bf980cb55d10e5e2416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4fd49304234bf980cb55d10e5e2416" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 35581 } } }
I20260619 06:45:11.933272 29867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:11.935658 29874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:45:11.936546 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:11.921062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56297 (local address 127.7.146.254:35581)
0619 06:45:11.921273 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:11.921280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:11.921520 (+ 240us) server_negotiation.cc:407] Connection header received
0619 06:45:11.921694 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:11.921698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:11.921761 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:11.921837 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:11.923244 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:11.924040 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:11.925314 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:11.925751 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:11.925857 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:11.928592 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:11.928599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:11.928604 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:11.928645 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:11.932503 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:11.933534 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:11.933540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:11.933544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:11.933766 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:11.934275 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:11.934281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:11.934284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:11.934721 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:11.935847 (+ 1126us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:11.935988 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 06:45:11.936196 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:45:11.940989 29874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:45:11.950462 29874 catalog_manager.cc:1347] Generated new cluster ID: f3f725db830845fd9675080ee42a58fa
I20260619 06:45:11.950575 29874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:45:11.962989 29874 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:45:11.964325 29874 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:45:11.971230 29874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d4fd49304234bf980cb55d10e5e2416: Generated new TSK 0
I20260619 06:45:11.972095 29874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:45:11.986560 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29789
I20260619 06:45:11.987733 29851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:45:12.077325 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 29789
2026-06-19T06:45:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1846 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:45:12 dist-test-slave-qfk4 krb5kdc[29888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:45:12 dist-test-slave-qfk4 krb5kdc[29888](info): set up 2 sockets
Jun 19 06:45:12 dist-test-slave-qfk4 krb5kdc[29888](info): commencing operation
krb5kdc: starting...
W20260619 06:45:13.092790 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:45:13 dist-test-slave-qfk4 krb5kdc[29888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:45:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:45:13.336510 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:36345
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:36345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:45:13.658460 29901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:13.658779 29901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:13.658913 29901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:13.666275 29901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:13.666414 29901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:13.666494 29901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:13.666585 29901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:13.666641 29901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:13.676756 29901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:36345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:36345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:13.678129 29901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:13.679417 29901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:13.692967 29909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:13.692939 29907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:13.692878 29906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:13.693897 29901 server_base.cc:1034] running on GCE node
I20260619 06:45:13.694756 29901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:13.696254 29901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:13.697458 29901 hybrid_clock.cc:648] HybridClock initialized: now 1781851513697417 us; error 42 us; skew 500 ppm
I20260619 06:45:13.697806 29901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:13 dist-test-slave-qfk4 krb5kdc[29888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851513, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:13.704520 29901 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:45:13.706288 29901 webserver.cc:460] Webserver started at http://127.7.146.254:38513/ using document root <none> and password file <none>
I20260619 06:45:13.706995 29901 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:13.707201 29901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:13.707545 29901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:13.710749 29901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "9dc36fa82def4937bfa9e784c4974d47"
format_stamp: "Formatted at 2026-06-19 06:45:13 on dist-test-slave-qfk4"
server_key: "27142cad1cce2422db189e9f3024cf56"
server_key_iv: "adb8ad0a8e55f359466afcd71cf7f907"
server_key_version: "encryptionkey@0"
I20260619 06:45:13.711570 29901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "9dc36fa82def4937bfa9e784c4974d47"
format_stamp: "Formatted at 2026-06-19 06:45:13 on dist-test-slave-qfk4"
server_key: "27142cad1cce2422db189e9f3024cf56"
server_key_iv: "adb8ad0a8e55f359466afcd71cf7f907"
server_key_version: "encryptionkey@0"
I20260619 06:45:13.716972 29901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 06:45:13.720412 29916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:13.721864 29901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:45:13.722040 29901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "9dc36fa82def4937bfa9e784c4974d47"
format_stamp: "Formatted at 2026-06-19 06:45:13 on dist-test-slave-qfk4"
server_key: "27142cad1cce2422db189e9f3024cf56"
server_key_iv: "adb8ad0a8e55f359466afcd71cf7f907"
server_key_version: "encryptionkey@0"
I20260619 06:45:13.722235 29901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:13.744704 29901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:13.748606 29901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:13.748868 29901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:13.766088 29901 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:36345
I20260619 06:45:13.766116 29967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:36345 every 8 connection(s)
I20260619 06:45:13.767431 29901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:45:13.769852 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 29901
I20260619 06:45:13.770025 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:45:13.770555 7755 external_mini_cluster.cc:1428] Setting key 0d3e068736e40e08f132b4b51a0ee57c
I20260619 06:45:13.771857 29968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:13.783639 29971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:13.772353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50259 (local address 127.7.146.254:36345)
0619 06:45:13.773121 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:13.773134 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:13.773173 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:45:13.774725 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:13.774749 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:13.775168 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:13.775502 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:13.776106 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:13.777469 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:13.779284 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:13.779954 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:13.780109 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:13.781314 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:13.781322 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:13.781340 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:13.781897 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:13.781970 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:13.781998 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:13.782100 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:13.782298 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 06:45:13.782557 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":353,"threads_started":1}
I20260619 06:45:13.784997 29968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47: Bootstrap starting.
I20260619 06:45:13.787333 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:45:13.788203 29968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:13.789229 29968 log.cc:824] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:13.792725 29968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47: No bootstrap required, opened a new log
I20260619 06:45:13.797801 29971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:13.791965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37309 (local address 127.7.146.254:36345)
0619 06:45:13.792113 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:13.792119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:13.792611 (+ 492us) server_negotiation.cc:407] Connection header received
0619 06:45:13.792802 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:13.792806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:13.792874 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:13.792959 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:13.793465 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:13.794188 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:13.795459 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:13.795930 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:13.796038 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:13.797079 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:13.797084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:13.797087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:13.797108 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:13.797147 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:13.797166 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:13.797246 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:13.797381 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 06:45:13.797478 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:45:13.801358 29968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc36fa82def4937bfa9e784c4974d47" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 36345 } }
I20260619 06:45:13.801781 29968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:45:13.801885 29968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc36fa82def4937bfa9e784c4974d47, State: Initialized, Role: FOLLOWER
I20260619 06:45:13.802482 29968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc36fa82def4937bfa9e784c4974d47" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 36345 } }
I20260619 06:45:13.802750 29968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:45:13.802865 29968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:45:13.803014 29968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:13.805744 29968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc36fa82def4937bfa9e784c4974d47" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 36345 } }
I20260619 06:45:13.806270 29968 leader_election.cc:302] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dc36fa82def4937bfa9e784c4974d47; no voters:
I20260619 06:45:13.806857 29968 leader_election.cc:288] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:45:13.806988 29978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:45:13.808115 29978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [term 1 LEADER]: Becoming Leader. State: Replica: 9dc36fa82def4937bfa9e784c4974d47, State: Running, Role: LEADER
I20260619 06:45:13.808681 29978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc36fa82def4937bfa9e784c4974d47" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 36345 } }
I20260619 06:45:13.809217 29968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:13.811362 29980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dc36fa82def4937bfa9e784c4974d47. Latest consensus state: current_term: 1 leader_uuid: "9dc36fa82def4937bfa9e784c4974d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc36fa82def4937bfa9e784c4974d47" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 36345 } } }
I20260619 06:45:13.811288 29979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dc36fa82def4937bfa9e784c4974d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc36fa82def4937bfa9e784c4974d47" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 36345 } } }
I20260619 06:45:13.811739 29979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:13.811739 29980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:13.813412 29987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:45:13.817739 29987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:45:13.826138 29987 catalog_manager.cc:1347] Generated new cluster ID: dba757315b8f47a09c5ad3486a05e99c
I20260619 06:45:13.826253 29987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:45:13.855571 29987 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:45:13.856995 29987 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:45:13.863843 29987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dc36fa82def4937bfa9e784c4974d47: Generated new TSK 0
I20260619 06:45:13.864809 29987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:45:13.877367 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29901
W20260619 06:45:13.878530 29963 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50259
W20260619 06:45:13.879324 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 29901: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:45:13.879530 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 29901
2026-06-19T06:45:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1802 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:45:13 dist-test-slave-qfk4 krb5kdc[29999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:45:13 dist-test-slave-qfk4 krb5kdc[29999](info): set up 2 sockets
Jun 19 06:45:13 dist-test-slave-qfk4 krb5kdc[29999](info): commencing operation
krb5kdc: starting...
W20260619 06:45:14.896958 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:45:15 dist-test-slave-qfk4 krb5kdc[29999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:45:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:45:15.141952 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:42279
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:42279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:45:15.461735 30012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:15.462055 30012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:15.462198 30012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:15.469933 30012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:15.470054 30012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:15.470132 30012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:15.470216 30012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:15.470276 30012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:15.480223 30012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:42279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:42279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:15.481586 30012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:15.482910 30012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:15.494706 30018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:15.494758 30020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:15.494654 30017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:15.495551 30012 server_base.cc:1034] running on GCE node
I20260619 06:45:15.496301 30012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:15.497731 30012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:15.498947 30012 hybrid_clock.cc:648] HybridClock initialized: now 1781851515498887 us; error 52 us; skew 500 ppm
I20260619 06:45:15.499343 30012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:15 dist-test-slave-qfk4 krb5kdc[29999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851515, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:15.502454 30012 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:45:15.503990 30012 webserver.cc:460] Webserver started at http://127.7.146.254:33619/ using document root <none> and password file <none>
I20260619 06:45:15.504689 30012 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:15.504807 30012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:15.505089 30012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:15.507985 30012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "cfc1a6ef2147477ba98bb56a4577b674"
format_stamp: "Formatted at 2026-06-19 06:45:15 on dist-test-slave-qfk4"
server_key: "f537e0668f0213570d223ded01833bc5"
server_key_iv: "c5e64bed3699efe5d523bad68af1d2ce"
server_key_version: "encryptionkey@0"
I20260619 06:45:15.508766 30012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "cfc1a6ef2147477ba98bb56a4577b674"
format_stamp: "Formatted at 2026-06-19 06:45:15 on dist-test-slave-qfk4"
server_key: "f537e0668f0213570d223ded01833bc5"
server_key_iv: "c5e64bed3699efe5d523bad68af1d2ce"
server_key_version: "encryptionkey@0"
I20260619 06:45:15.513904 30012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 06:45:15.517400 30027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:15.518781 30012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:45:15.519135 30012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "cfc1a6ef2147477ba98bb56a4577b674"
format_stamp: "Formatted at 2026-06-19 06:45:15 on dist-test-slave-qfk4"
server_key: "f537e0668f0213570d223ded01833bc5"
server_key_iv: "c5e64bed3699efe5d523bad68af1d2ce"
server_key_version: "encryptionkey@0"
I20260619 06:45:15.519335 30012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:15.525303 30012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:15.529352 30012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:15.529621 30012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:15.546470 30012 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:42279
I20260619 06:45:15.546501 30078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:42279 every 8 connection(s)
I20260619 06:45:15.547814 30012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:45:15.551600 30079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:15.555054 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 30012
I20260619 06:45:15.555231 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:45:15.555747 7755 external_mini_cluster.cc:1428] Setting key df1dca4ca528397d270817c72ba911ef
I20260619 06:45:15.563287 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:15.557576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42323 (local address 127.7.146.254:42279)
0619 06:45:15.558323 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:15.558335 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:15.558376 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:45:15.559995 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:15.560018 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:15.560386 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:15.560788 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:15.561768 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:15.561774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:15.561791 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:15.561807 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:15.561865 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:15.561884 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:15.562066 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:15.562161 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 06:45:15.562333 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":327,"threads_started":1}
I20260619 06:45:15.563932 30079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674: Bootstrap starting.
I20260619 06:45:15.567248 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:45:15.567420 30079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:15.568635 30079 log.cc:824] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:15.571862 30079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674: No bootstrap required, opened a new log
I20260619 06:45:15.574633 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:15.572231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58211 (local address 127.7.146.254:42279)
0619 06:45:15.572414 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:15.572420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:15.572827 (+ 407us) server_negotiation.cc:407] Connection header received
0619 06:45:15.572951 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:15.572956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:15.573014 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:15.573126 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:15.574028 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:15.574032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:15.574035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:45:15.574050 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:15.574093 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:45:15.574108 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:15.574202 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:15.574299 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 06:45:15.574349 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:45:15.580281 30079 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc1a6ef2147477ba98bb56a4577b674" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42279 } }
I20260619 06:45:15.580746 30079 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:45:15.580852 30079 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc1a6ef2147477ba98bb56a4577b674, State: Initialized, Role: FOLLOWER
I20260619 06:45:15.581470 30079 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc1a6ef2147477ba98bb56a4577b674" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42279 } }
I20260619 06:45:15.581682 30079 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:45:15.581811 30079 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:45:15.581948 30079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:15.584160 30079 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc1a6ef2147477ba98bb56a4577b674" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42279 } }
I20260619 06:45:15.584755 30079 leader_election.cc:302] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfc1a6ef2147477ba98bb56a4577b674; no voters:
I20260619 06:45:15.585321 30079 leader_election.cc:288] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:45:15.585449 30089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:45:15.586032 30089 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [term 1 LEADER]: Becoming Leader. State: Replica: cfc1a6ef2147477ba98bb56a4577b674, State: Running, Role: LEADER
I20260619 06:45:15.586534 30089 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc1a6ef2147477ba98bb56a4577b674" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42279 } }
I20260619 06:45:15.587288 30079 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:15.589351 30091 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfc1a6ef2147477ba98bb56a4577b674. Latest consensus state: current_term: 1 leader_uuid: "cfc1a6ef2147477ba98bb56a4577b674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc1a6ef2147477ba98bb56a4577b674" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42279 } } }
I20260619 06:45:15.589385 30090 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfc1a6ef2147477ba98bb56a4577b674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc1a6ef2147477ba98bb56a4577b674" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 42279 } } }
I20260619 06:45:15.589769 30091 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:15.589779 30090 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:15.591267 30095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:45:15.595932 30095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:45:15.603823 30095 catalog_manager.cc:1347] Generated new cluster ID: 32ec9400c3a949f3af11d4974f5028f0
I20260619 06:45:15.603942 30095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:45:15.618554 30095 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:45:15.619033 30095 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:45:15.628669 30095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfc1a6ef2147477ba98bb56a4577b674: Generated new TSK 0
I20260619 06:45:15.629594 30095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:45:15.641389 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 30012
W20260619 06:45:15.642637 30074 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42323
W20260619 06:45:15.643362 7755 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 30012: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:45:15.643514 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 30012
2026-06-19T06:45:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1762 ms)
[----------] 18 tests from AuthTokenIssuingTest (32825 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:45:15 dist-test-slave-qfk4 krb5kdc[30110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:45:15 dist-test-slave-qfk4 krb5kdc[30110](info): set up 2 sockets
Jun 19 06:45:15 dist-test-slave-qfk4 krb5kdc[30110](info): commencing operation
krb5kdc: starting...
W20260619 06:45:16.661101 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:45:16 dist-test-slave-qfk4 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:45:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:45:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:45:16.906507 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38477
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:38477,127.7.146.253:36549,127.7.146.252:33421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:45:17.229185 30123 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:45:17.229543 30123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:17.229640 30123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:17.229751 30123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:17.237124 30123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:17.237242 30123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:17.237324 30123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:17.237390 30123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:17.237473 30123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:17.247471 30123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:38477,127.7.146.253:36549,127.7.146.252:33421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:17.248816 30123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:17.250085 30123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:17.261456 30129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:17.261404 30131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:17.261435 30128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:17.262261 30123 server_base.cc:1034] running on GCE node
I20260619 06:45:17.263031 30123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:17.264390 30123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:17.265604 30123 hybrid_clock.cc:648] HybridClock initialized: now 1781851517265529 us; error 48 us; skew 500 ppm
I20260619 06:45:17.265928 30123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:17 dist-test-slave-qfk4 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851517, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:17.268968 30123 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:45:17.270552 30123 webserver.cc:460] Webserver started at http://127.7.146.254:41767/ using document root <none> and password file <none>
I20260619 06:45:17.271224 30123 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:17.271306 30123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:17.271587 30123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:17.274565 30123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "0eac3b29cde442f6b57bea5850c0bb96"
format_stamp: "Formatted at 2026-06-19 06:45:17 on dist-test-slave-qfk4"
server_key: "550a883b80817ce5b28c8b4c86659231"
server_key_iv: "00d1eda2c5756a120085c5b219aeb82e"
server_key_version: "encryptionkey@0"
I20260619 06:45:17.275379 30123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "0eac3b29cde442f6b57bea5850c0bb96"
format_stamp: "Formatted at 2026-06-19 06:45:17 on dist-test-slave-qfk4"
server_key: "550a883b80817ce5b28c8b4c86659231"
server_key_iv: "00d1eda2c5756a120085c5b219aeb82e"
server_key_version: "encryptionkey@0"
I20260619 06:45:17.280601 30123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 06:45:17.284114 30138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:17.285583 30123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:45:17.285763 30123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "0eac3b29cde442f6b57bea5850c0bb96"
format_stamp: "Formatted at 2026-06-19 06:45:17 on dist-test-slave-qfk4"
server_key: "550a883b80817ce5b28c8b4c86659231"
server_key_iv: "00d1eda2c5756a120085c5b219aeb82e"
server_key_version: "encryptionkey@0"
I20260619 06:45:17.285964 30123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:17.302873 30123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:17.307413 30123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:17.307662 30123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:17.324926 30123 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:38477
I20260619 06:45:17.324945 30189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:38477 every 8 connection(s)
I20260619 06:45:17.326328 30123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:45:17.330008 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 30123
I20260619 06:45:17.330150 30190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:17.330192 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:45:17.330777 7755 external_mini_cluster.cc:1428] Setting key 7f20a211aaab56cf98a6a166ac4fb81b
I20260619 06:45:17.336766 30190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:45:17 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851517, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.253@KRBTEST.COM" created.
I20260619 06:45:17.358531 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.340026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.7.146.254:38477)
0619 06:45:17.343387 (+ 3361us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:17.343392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:17.343419 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:45:17.343546 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:17.343551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:17.343773 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:17.343944 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:17.345361 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.346510 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.347819 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.348354 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.348515 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:17.351566 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:17.351573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:17.351585 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:17.351616 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:17.354470 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:17.355090 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:17.355096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:17.355098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:17.355243 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:17.355587 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:17.355590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:17.355592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:17.356052 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:17.356233 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:17.356778 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 06:45:17.357115 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":3214,"thread_start_us":3029,"threads_started":1}
I20260619 06:45:17.358547 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.340026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38477 (local address 127.0.0.1:40364)
0619 06:45:17.340873 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.340920 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:17.342781 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:17.344002 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:17.344009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:17.344747 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:17.345162 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:17.345183 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.346649 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.346652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:17.347686 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:17.347707 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.348595 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.348602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:17.348767 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:17.349337 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:17.349359 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:17.351375 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:17.354576 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:17.354587 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:17.354600 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:17.354968 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:17.355339 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:17.355344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:17.355348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:17.355486 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:17.356182 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:17.356190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:17.356394 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 06:45:17.356624 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 06:45:17.357234 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":139,"thread_start_us":341,"threads_started":1}
I20260619 06:45:17.361681 30190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:45:17.362709 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.362292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34052)
0619 06:45:17.362470 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.362588 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 06:45:17.362901 30140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
I20260619 06:45:17.364342 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.364010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34058)
0619 06:45:17.364143 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.364214 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 06:45:17.364512 30190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:36549: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab.
Entry for principal kudu/127.7.146.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab.
I20260619 06:45:17.384034 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.253:36549
--webserver_interface=127.7.146.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab
--principal=kudu/127.7.146.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:38477,127.7.146.253:36549,127.7.146.252:33421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:45:17.413861 30190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } attempt: 1
I20260619 06:45:17.414953 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.414511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34070)
0619 06:45:17.414703 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.414825 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 06:45:17.415954 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.415650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34084)
0619 06:45:17.415768 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.415834 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 06:45:17.416137 30190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:36549: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
I20260619 06:45:17.484382 30190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } attempt: 2
I20260619 06:45:17.485680 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.485196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34090)
0619 06:45:17.485447 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.485547 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 06:45:17.486644 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.486323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34092)
0619 06:45:17.486446 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.486514 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 06:45:17.486843 30190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:36549: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
I20260619 06:45:17.578092 30190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } attempt: 3
I20260619 06:45:17.579309 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.578839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34094)
0619 06:45:17.579069 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.579167 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 06:45:17.580246 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.579915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34102)
0619 06:45:17.580048 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.580113 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 06:45:17.580406 30190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:36549: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
W20260619 06:45:17.702420 30197 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:45:17.702785 30197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:17.702886 30197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:17.703013 30197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:17.710379 30197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:17.710487 30197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:17.710577 30197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:17.710642 30197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:17.710726 30197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:17.720798 30197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:38477,127.7.146.253:36549,127.7.146.252:33421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab
--principal=kudu/127.7.146.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.253:36549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.146.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:17.722090 30197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:17.723425 30197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:45:17.723637 30190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } attempt: 4
I20260619 06:45:17.724745 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.724207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34112)
0619 06:45:17.724479 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.724598 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260619 06:45:17.725745 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.725406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34120)
0619 06:45:17.725549 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.725625 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:45:17.725907 30190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:36549: Network error: Client connection negotiation failed: client connection to 127.7.146.253:36549: connect: Connection refused (error 111)
W20260619 06:45:17.735098 30202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:17.735030 30205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:17.735041 30203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:17.735608 30197 server_base.cc:1034] running on GCE node
I20260619 06:45:17.736460 30197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:17.737834 30197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:17.739044 30197 hybrid_clock.cc:648] HybridClock initialized: now 1781851517738984 us; error 38 us; skew 500 ppm
I20260619 06:45:17.739355 30197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:17 dist-test-slave-qfk4 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851517, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:17.742357 30197 init.cc:377] Logged in from keytab as kudu/127.7.146.253@KRBTEST.COM (short username kudu)
I20260619 06:45:17.743752 30197 webserver.cc:460] Webserver started at http://127.7.146.253:44831/ using document root <none> and password file <none>
I20260619 06:45:17.744400 30197 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:17.744495 30197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:17.744822 30197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:17.747782 30197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/data/instance:
uuid: "b0a901906fef4c24b0652387f79bea31"
format_stamp: "Formatted at 2026-06-19 06:45:17 on dist-test-slave-qfk4"
server_key: "fb074982bc3380978d96988cd6072b34"
server_key_iv: "dd5b5495f25a73a434ceb220fb074ca1"
server_key_version: "encryptionkey@0"
I20260619 06:45:17.748528 30197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/wal/instance:
uuid: "b0a901906fef4c24b0652387f79bea31"
format_stamp: "Formatted at 2026-06-19 06:45:17 on dist-test-slave-qfk4"
server_key: "fb074982bc3380978d96988cd6072b34"
server_key_iv: "dd5b5495f25a73a434ceb220fb074ca1"
server_key_version: "encryptionkey@0"
I20260619 06:45:17.753859 30197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 06:45:17.757359 30212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:17.758860 30197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:45:17.759043 30197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/wal
uuid: "b0a901906fef4c24b0652387f79bea31"
format_stamp: "Formatted at 2026-06-19 06:45:17 on dist-test-slave-qfk4"
server_key: "fb074982bc3380978d96988cd6072b34"
server_key_iv: "dd5b5495f25a73a434ceb220fb074ca1"
server_key_version: "encryptionkey@0"
I20260619 06:45:17.759260 30197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:17.780257 30197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:17.784694 30197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:17.784948 30197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:17.801625 30197 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.253:36549
I20260619 06:45:17.801648 30263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.253:36549 every 8 connection(s)
I20260619 06:45:17.802942 30197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
I20260619 06:45:17.806521 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 30197
I20260619 06:45:17.806623 30264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:17.806707 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-1/wal/instance
I20260619 06:45:17.807235 7755 external_mini_cluster.cc:1428] Setting key d12d63a89619aabda7bcb2a6fc2d011e
I20260619 06:45:17.815049 30264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:45:17 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851517, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.252@KRBTEST.COM" created.
I20260619 06:45:17.835351 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.818127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.7.146.254:38477)
0619 06:45:17.818318 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:17.818323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:17.819729 (+ 1406us) server_negotiation.cc:407] Connection header received
0619 06:45:17.821653 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:17.821659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:17.821755 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:17.821845 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:17.823361 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.824033 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.825583 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.825983 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.826090 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:17.829657 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:17.829662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:17.829666 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:17.829698 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:17.832276 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:17.833047 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:17.833050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:17.833053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:17.833202 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:17.833619 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:17.833622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:17.833625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:17.833920 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:17.834052 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:17.834794 (+ 742us) server_negotiation.cc:299] Negotiation successful
0619 06:45:17.835024 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:45:17.836346 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.818789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38477 (local address 127.0.0.1:40378)
0619 06:45:17.819563 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.819605 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:17.821365 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:17.822091 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:17.822100 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:17.822619 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:17.823181 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:17.823209 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.824188 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.824199 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:17.825438 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:17.825450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.826201 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.826208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:17.826430 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:17.827083 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:17.827105 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:17.829437 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:17.832405 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:17.832415 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:17.832432 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:17.832908 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:17.833303 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:17.833308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:17.833312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:17.833513 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:17.834040 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:17.834048 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:17.834427 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 06:45:17.834666 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 06:45:17.835186 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":367,"threads_started":1}
I20260619 06:45:17.838142 30264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:45:17 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851517, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
I20260619 06:45:17.852361 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.838938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34124)
0619 06:45:17.839090 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.839105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:17.839191 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:17.840178 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:17.840183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:17.840633 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:17.840788 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:17.840797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.842085 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.842089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:17.842914 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:17.842925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.843709 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.843715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:17.843861 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:17.844415 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:17.844431 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:17.846548 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:17.850176 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:17.850179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:17.850182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:17.850536 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:17.850905 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:17.850908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:17.850910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:17.851030 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:17.851687 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:17.851690 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:17.851801 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 06:45:17.851902 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:45:17.852094 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 06:45:17.852730 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.839208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.7.146.253:36549)
0619 06:45:17.839775 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:17.839781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:17.839806 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:45:17.839849 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:17.839855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:17.840042 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:17.840170 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:17.840936 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.841960 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.843032 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:17.843557 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:17.843661 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:17.846764 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:17.846777 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:17.846794 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:17.846833 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:17.850038 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:17.850658 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:17.850667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:17.850670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:17.850817 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:17.851133 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:17.851138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:17.851142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:17.851578 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:17.851792 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:17.852065 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 06:45:17.852307 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":305,"threads_started":1}
I20260619 06:45:17.855242 30264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:45:17.856402 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.855945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36188)
0619 06:45:17.856111 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.856260 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 06:45:17.856918 30213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
I20260619 06:45:17.858397 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.858025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36194)
0619 06:45:17.858211 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.858293 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:45:17.858603 30264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:33421: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab.
Entry for principal kudu/127.7.146.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab.
I20260619 06:45:17.865891 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.252:33421
--webserver_interface=127.7.146.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab
--principal=kudu/127.7.146.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:38477,127.7.146.253:36549,127.7.146.252:33421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:45:17.907933 30264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } attempt: 1
I20260619 06:45:17.909189 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.908721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36206)
0619 06:45:17.908899 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.909004 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 06:45:17.910220 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.909882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36210)
0619 06:45:17.910032 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.910107 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:45:17.910439 30264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:33421: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
I20260619 06:45:17.978648 30264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } attempt: 2
I20260619 06:45:17.979748 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.979289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36224)
0619 06:45:17.979489 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.979588 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 06:45:17.980791 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:17.980424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36232)
0619 06:45:17.980549 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:17.980663 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:45:17.981026 30264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:33421: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
I20260619 06:45:18.025158 30190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } attempt: 5
Jun 19 06:45:18 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851517, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
I20260619 06:45:18.039029 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.026091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.7.146.253:36549)
0619 06:45:18.026309 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.026316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.026343 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:45:18.026469 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.026476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.026588 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.026729 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.027492 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.028650 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.029830 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.030498 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.030636 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.033892 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.033897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.033900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.033934 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.036460 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.037262 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.037266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.037269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.037385 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.037760 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.037763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.037765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.038027 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.038136 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.038615 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.038803 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:45:18.039162 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.025921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34126)
0619 06:45:18.026235 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.026253 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:18.026365 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:18.026744 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:18.026747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:18.027105 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:18.027322 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.027332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.028810 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.028814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.029671 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.029682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.030650 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.030654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.030814 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.031425 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:18.031443 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:18.033679 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:18.036626 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.036632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.036636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.037154 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.037487 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.037492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.037495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.037663 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.038134 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:18.038140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:18.038337 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 06:45:18.038516 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 06:45:18.038816 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260619 06:45:18.040316 30190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:45:18.041539 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.041065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36234)
0619 06:45:18.041253 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.041359 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:45:18.042523 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.042128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36244)
0619 06:45:18.042284 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.042390 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 06:45:18.042794 30190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:33421: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
I20260619 06:45:18.072245 30264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } attempt: 3
I20260619 06:45:18.073438 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.072998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36250)
0619 06:45:18.073186 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.073280 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 06:45:18.074488 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.074080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36254)
0619 06:45:18.074294 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.074372 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260619 06:45:18.074712 30264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:33421: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
I20260619 06:45:18.094051 30190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } attempt: 1
I20260619 06:45:18.095355 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.094803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36262)
0619 06:45:18.095024 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.095148 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 06:45:18.096436 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.096065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36274)
0619 06:45:18.096216 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.096306 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:45:18.096674 30190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:33421: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
I20260619 06:45:18.164891 30190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } attempt: 2
I20260619 06:45:18.166290 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.165632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36286)
0619 06:45:18.165950 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.166069 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 06:45:18.167507 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.167058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36302)
0619 06:45:18.167248 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.167365 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:45:18.167872 30190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:33421: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
W20260619 06:45:18.202883 30271 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:45:18.203214 30271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:18.203284 30271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:18.203363 30271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:18.210666 30271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:18.210759 30271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:18.210822 30271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:18.210875 30271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:18.210934 30271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:18.217924 30264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } attempt: 4
I20260619 06:45:18.219169 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.218670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36318)
0619 06:45:18.218886 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.218978 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 06:45:18.220228 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.219885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36320)
0619 06:45:18.220031 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.220100 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:45:18.220414 30264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:33421: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
I20260619 06:45:18.221500 30271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:38477,127.7.146.253:36549,127.7.146.252:33421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab
--principal=kudu/127.7.146.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.252:33421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.146.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:18.222743 30271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:18.223980 30271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:18.235775 30276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:18.235962 30277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:18.235756 30279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:18.236429 30271 server_base.cc:1034] running on GCE node
I20260619 06:45:18.237265 30271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:18.238688 30271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:18.239934 30271 hybrid_clock.cc:648] HybridClock initialized: now 1781851518239857 us; error 52 us; skew 500 ppm
I20260619 06:45:18.240284 30271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:18 dist-test-slave-qfk4 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851518, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:18.243372 30271 init.cc:377] Logged in from keytab as kudu/127.7.146.252@KRBTEST.COM (short username kudu)
I20260619 06:45:18.244791 30271 webserver.cc:460] Webserver started at http://127.7.146.252:35853/ using document root <none> and password file <none>
I20260619 06:45:18.245543 30271 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:18.245633 30271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:18.245903 30271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:18.248986 30271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/data/instance:
uuid: "5fb5bf9566b94f7e8f92b74b447af8a2"
format_stamp: "Formatted at 2026-06-19 06:45:18 on dist-test-slave-qfk4"
server_key: "f7b9d77cdd5309b988760fb6ab292383"
server_key_iv: "a484ff8bf4ee6d5737250152b84c6db8"
server_key_version: "encryptionkey@0"
I20260619 06:45:18.249712 30271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/wal/instance:
uuid: "5fb5bf9566b94f7e8f92b74b447af8a2"
format_stamp: "Formatted at 2026-06-19 06:45:18 on dist-test-slave-qfk4"
server_key: "f7b9d77cdd5309b988760fb6ab292383"
server_key_iv: "a484ff8bf4ee6d5737250152b84c6db8"
server_key_version: "encryptionkey@0"
I20260619 06:45:18.254797 30271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:45:18.258375 30286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:18.259754 30271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:45:18.259908 30271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/wal
uuid: "5fb5bf9566b94f7e8f92b74b447af8a2"
format_stamp: "Formatted at 2026-06-19 06:45:18 on dist-test-slave-qfk4"
server_key: "f7b9d77cdd5309b988760fb6ab292383"
server_key_iv: "a484ff8bf4ee6d5737250152b84c6db8"
server_key_version: "encryptionkey@0"
I20260619 06:45:18.260138 30271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:18.274102 30190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } attempt: 3
I20260619 06:45:18.275228 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.274723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36332)
0619 06:45:18.274902 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.275001 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:45:18.276180 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.275828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36342)
0619 06:45:18.276006 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.276085 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 06:45:18.276367 30190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:33421: Network error: Client connection negotiation failed: client connection to 127.7.146.252:33421: connect: Connection refused (error 111)
I20260619 06:45:18.286187 30271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:18.290251 30271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:18.290483 30271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:18.306928 30271 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.252:33421
I20260619 06:45:18.306949 30337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.252:33421 every 8 connection(s)
I20260619 06:45:18.308256 30271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
I20260619 06:45:18.310253 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 30271
I20260619 06:45:18.310428 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851174422545-7755-0/minicluster-data/master-2/wal/instance
I20260619 06:45:18.310962 7755 external_mini_cluster.cc:1428] Setting key dd93fd56f7792393a25c259c810309a9
I20260619 06:45:18.312451 30338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:18.320103 30338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:45:18 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
Jun 19 06:45:18 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851518, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:45:18.334537 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.313070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36350 (local address 127.7.146.252:33421)
0619 06:45:18.313840 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.313852 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.313892 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:45:18.315443 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.315465 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.315862 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.316224 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.317640 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.318913 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.320157 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.320892 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.321065 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.325892 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.325902 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.325916 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.325961 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.329607 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.330485 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.330497 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.330504 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.330717 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.331185 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.331190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.331193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.331871 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.332195 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.332700 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.333289 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":333,"threads_started":1}
I20260619 06:45:18.339143 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.322652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40380 (local address 127.7.146.254:38477)
0619 06:45:18.322800 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.322804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.323602 (+ 798us) server_negotiation.cc:407] Connection header received
0619 06:45:18.323811 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.323819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.323896 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.323990 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.325338 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.326266 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.328251 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.329045 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.329205 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.332866 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.332874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.332880 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.332927 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.336267 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.337061 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.337066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.337069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.337213 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.337699 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.337704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.337707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.338013 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.338179 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.338687 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.338898 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:45:18.339316 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.322689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38477 (local address 127.0.0.1:40380)
0619 06:45:18.323439 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.323475 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:18.323665 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:18.324125 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:18.324134 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:18.324899 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:18.325176 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.325186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.326433 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.326439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.328096 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.328111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.329241 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.329251 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.329446 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.330057 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:18.330077 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:18.332620 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:18.336403 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.336415 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.336423 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.336917 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.337323 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.337328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.337331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.337508 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.338170 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:18.338178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:18.338381 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 06:45:18.338534 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 06:45:18.338853 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":390,"threads_started":1}
I20260619 06:45:18.340679 30338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:45:18 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
Jun 19 06:45:18 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851518, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
I20260619 06:45:18.352734 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.337813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.7.146.254:38477)
0619 06:45:18.338377 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.338381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.338397 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:45:18.338432 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.338435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.338485 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.338613 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.340297 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.341440 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.342636 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.343030 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.343134 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.347263 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.347268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.347272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.347304 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.349992 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.350759 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.350762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.350765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.350881 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.351251 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.351256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.351259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.351538 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.351779 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.352061 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.352272 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":313,"threads_started":1}
I20260619 06:45:18.353085 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.341520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:36549 (local address 127.0.0.1:34136)
0619 06:45:18.341754 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.341770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:18.341865 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:18.342141 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:18.342145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:18.342493 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:18.342624 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.342632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.343557 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.343562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.344422 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.344433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.345121 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.345127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.345289 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.345851 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:18.345874 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:18.347904 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:18.350887 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.350893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.350897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.351254 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.351588 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.351593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.351597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.351733 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.352192 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:18.352205 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:18.352344 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 06:45:18.352511 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 06:45:18.352772 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260619 06:45:18.353133 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.341515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34136 (local address 127.7.146.253:36549)
0619 06:45:18.341691 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.341721 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.341830 (+ 109us) server_negotiation.cc:407] Connection header received
0619 06:45:18.341968 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.341972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.342032 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.342110 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.342760 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.343434 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.344551 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.344987 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.345091 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.348085 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.348090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.348093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.348126 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.350771 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.351357 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.351362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.351366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.351490 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.351827 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.351832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.351836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.352079 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.352207 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.352654 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.352864 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:45:18.354652 30338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:45:18 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851518, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
Jun 19 06:45:18 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
I20260619 06:45:18.367286 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.355394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36360)
0619 06:45:18.355532 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.355548 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:18.355638 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:18.356053 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:18.356056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:18.356499 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:18.356716 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.356725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.357588 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.357593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.358454 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.358465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.359111 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.359117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.359291 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.359833 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:18.359847 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:18.361918 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:18.364917 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.364922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.364925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.365386 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.365727 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.365733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.365737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.365896 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.366368 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:18.366373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:18.366497 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 06:45:18.366620 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 06:45:18.366896 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 06:45:18.367430 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.355390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.7.146.252:33421)
0619 06:45:18.355543 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.355548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.355595 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:45:18.355750 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.355755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.355872 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.356000 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.356834 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.357479 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.358575 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.358968 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.359074 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.362092 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.362098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.362103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.362137 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.364803 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.365489 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.365493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.365495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.365606 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.366002 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.366005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.366007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.366261 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.366381 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.366738 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.367090 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":60}
I20260619 06:45:18.368795 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.354641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.7.146.253:36549)
0619 06:45:18.354799 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.354804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.354820 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:45:18.354955 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.354960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.355028 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.355116 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.356631 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.357333 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.358476 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.358868 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.358970 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.363099 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.363106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.363111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.363151 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.366195 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.366793 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.366799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.366802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.366932 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.367289 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.367294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.367298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.367620 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.367879 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.368135 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.368345 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:45:18.372514 30338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2: Bootstrap starting.
I20260619 06:45:18.373597 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:45:18.375718 30338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:18.377008 30338 log.cc:824] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:18.376188 30165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0eac3b29cde442f6b57bea5850c0bb96"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40384
I20260619 06:45:18.380128 30338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2: No bootstrap required, opened a new log
I20260619 06:45:18.390348 30338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } }
I20260619 06:45:18.390674 30338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb5bf9566b94f7e8f92b74b447af8a2, State: Initialized, Role: FOLLOWER
I20260619 06:45:18.391177 30338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } }
I20260619 06:45:18.392117 30345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } } }
I20260619 06:45:18.392414 30345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:18.393350 30338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:45:18.402657 30356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:45:18.402798 30356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:45:18.400427 30271 master_runner.cc:186] Error getting master registration for 127.7.146.254:38477: OK, instance_id { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" instance_seqno: 1781851517320088 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:45:18.408479 30271 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:45:18.453553 30190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } attempt: 4
Jun 19 06:45:18 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851517, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
I20260619 06:45:18.470223 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.454367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36372)
0619 06:45:18.454552 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.454571 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:18.454688 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:18.455057 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:18.455062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:18.455586 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:18.455808 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.455822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.457323 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.457329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.458873 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.458893 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.459955 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.459961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.460171 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.460853 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:18.460872 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:18.463188 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:18.467386 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.467393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.467397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.467940 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.468360 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.468365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.468369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.468535 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.469179 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:18.469184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:18.469370 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 06:45:18.469515 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 06:45:18.469817 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:45:18.470268 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.454430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.7.146.252:33421)
0619 06:45:18.454641 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.454647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.454674 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:45:18.454815 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.454820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.454898 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.455006 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.455990 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.457177 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.459059 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.459792 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.459940 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.463406 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.463413 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.463419 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.463517 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.467218 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.468070 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.468076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.468080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.468252 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.468651 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.468657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.468660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.469071 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.469248 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.469634 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.469863 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:45:18.476310 30190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96: Bootstrap starting.
I20260619 06:45:18.480708 30190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:18.482113 30190 log.cc:824] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:18.485755 30190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96: No bootstrap required, opened a new log
I20260619 06:45:18.493515 30190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } }
I20260619 06:45:18.493952 30190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eac3b29cde442f6b57bea5850c0bb96, State: Initialized, Role: FOLLOWER
I20260619 06:45:18.494496 30190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } }
I20260619 06:45:18.495528 30359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } } }
I20260619 06:45:18.495952 30359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:18.496836 30190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:18.496939 30165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 06:45:18.497138 30165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:18.499593 30165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } }
I20260619 06:45:18.500934 30165 leader_election.cc:288] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [CANDIDATE]: Term 1 election: Requested vote from peers b0a901906fef4c24b0652387f79bea31 (127.7.146.253:36549), 5fb5bf9566b94f7e8f92b74b447af8a2 (127.7.146.252:33421)
I20260619 06:45:18.502341 30313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0eac3b29cde442f6b57bea5850c0bb96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2"
I20260619 06:45:18.502820 30313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:18.506646 30313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eac3b29cde442f6b57bea5850c0bb96 in term 1.
I20260619 06:45:18.507686 30139 leader_election.cc:302] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eac3b29cde442f6b57bea5850c0bb96, 5fb5bf9566b94f7e8f92b74b447af8a2; no voters:
I20260619 06:45:18.508244 30359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:45:18.519580 30264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } attempt: 5
Jun 19 06:45:18 dist-test-slave-qfk4 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851517, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
W20260619 06:45:18.542495 30292 tablet.cc:2369] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:45:18.550148 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.521336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.7.146.252:33421)
0619 06:45:18.521529 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.521536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.521555 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:45:18.521626 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.521631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.521716 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.521819 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.523039 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.524229 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.525976 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.526649 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.526786 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.537164 (+ 10378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.537171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.537176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.537222 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.541101 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.544416 (+ 3315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.544426 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.544429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.544757 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.546180 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.546187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.546190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.546799 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.547058 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.549031 (+ 1973us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.549479 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:45:18.554493 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.526093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.7.146.253:36549)
0619 06:45:18.526258 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.526263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.536077 (+ 9814us) server_negotiation.cc:407] Connection header received
0619 06:45:18.536612 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.536617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.536706 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.536780 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.539643 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.540780 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.543115 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.543823 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.543923 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.547578 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.547585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.547589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.547641 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.551077 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.551996 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.552001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.552004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.552145 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.552632 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.552638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.552641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.553000 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.553220 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.553779 (+ 559us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.554014 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":42880}
I20260619 06:45:18.557547 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.520407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:33421 (local address 127.0.0.1:36388)
0619 06:45:18.520923 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:18.520946 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:18.521087 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:18.522036 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:18.522041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:18.522612 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:18.522840 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.522853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.524411 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.524418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.525748 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:18.525766 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.527039 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.527046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:18.527275 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.528062 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:18.528088 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:18.536634 (+ 8546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:18.541571 (+ 4937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.541578 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.541582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.544030 (+ 2448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.545081 (+ 1051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:18.545086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:18.545090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:18.545804 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:18.547273 (+ 1469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:18.547307 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:18.548236 (+ 929us) client_negotiation.cc:769] Sending connection context
0619 06:45:18.550759 (+ 2523us) client_negotiation.cc:240] Negotiation successful
0619 06:45:18.553150 (+ 2391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366}
W20260619 06:45:18.560882 30144 tablet.cc:2369] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:45:18.565842 30264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31: Bootstrap starting.
I20260619 06:45:18.570331 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.532624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.7.146.254:38477)
0619 06:45:18.532811 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.532818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.546768 (+ 13950us) server_negotiation.cc:407] Connection header received
0619 06:45:18.547300 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.547306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.547389 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.547458 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.549249 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.550328 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.553040 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.553760 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.553858 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.559296 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.559303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.559308 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.559356 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.564201 (+ 4845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.567101 (+ 2900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.567107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.567110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.567315 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.568072 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.568078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.568081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.568449 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.568746 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.569654 (+ 908us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.569911 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:45:18.570366 30264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:18.573326 30264 log.cc:824] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:18.582163 30264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31: No bootstrap required, opened a new log
I20260619 06:45:18.585546 30359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [term 1 LEADER]: Becoming Leader. State: Replica: 0eac3b29cde442f6b57bea5850c0bb96, State: Running, Role: LEADER
I20260619 06:45:18.586236 30359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } }
I20260619 06:45:18.591462 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:18.530712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.7.146.252:33421)
0619 06:45:18.552138 (+ 21426us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.552144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.552342 (+ 198us) server_negotiation.cc:407] Connection header received
0619 06:45:18.552990 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.552997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.553801 (+ 804us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.555100 (+ 1299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.556519 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.559224 (+ 2705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.575091 (+ 15867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:18.575869 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:18.576011 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:18.579383 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:18.579391 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:18.579396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:18.579465 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:18.585750 (+ 6285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.586895 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.586900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.586903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.587161 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:18.587762 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:18.587767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:18.587770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:18.588416 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:18.589949 (+ 1533us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:18.590222 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 06:45:18.590747 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20309,"thread_start_us":47547,"threads_started":1}
I20260619 06:45:18.593026 30364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eac3b29cde442f6b57bea5850c0bb96. Latest consensus state: current_term: 1 leader_uuid: "0eac3b29cde442f6b57bea5850c0bb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } } }
I20260619 06:45:18.593591 30364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:18.594882 30382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:45:18.602993 30264 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } }
I20260619 06:45:18.603406 30264 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a901906fef4c24b0652387f79bea31, State: Initialized, Role: FOLLOWER
I20260619 06:45:18.603943 30264 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } }
I20260619 06:45:18.605427 30384 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } } }
I20260619 06:45:18.606459 30264 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:18.606559 30239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0eac3b29cde442f6b57bea5850c0bb96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b0a901906fef4c24b0652387f79bea31"
I20260619 06:45:18.606992 30239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:18.610726 30239 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eac3b29cde442f6b57bea5850c0bb96 in term 1.
I20260619 06:45:18.606355 30384 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:18.615701 30382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:45:18.600392 30123 master_runner.cc:186] Error getting master registration for 127.7.146.253:36549: OK, instance_id { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" instance_seqno: 1781851517796887 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:45:18.617156 30123 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:45:18.626607 30239 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [term 1 FOLLOWER]: Refusing update from remote peer 0eac3b29cde442f6b57bea5850c0bb96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:45:18.627668 30313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [term 1 FOLLOWER]: Refusing update from remote peer 0eac3b29cde442f6b57bea5850c0bb96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:45:18.628067 30364 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:45:18.629748 30364 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:45:18.650522 30345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eac3b29cde442f6b57bea5850c0bb96. Latest consensus state: current_term: 1 leader_uuid: "0eac3b29cde442f6b57bea5850c0bb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } } }
I20260619 06:45:18.650869 30345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:18.655346 30384 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eac3b29cde442f6b57bea5850c0bb96. Latest consensus state: current_term: 1 leader_uuid: "0eac3b29cde442f6b57bea5850c0bb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } } }
I20260619 06:45:18.655678 30384 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:18.656657 30389 mvcc.cc:205] Tried to move back new op lower bound from 7298463820282531840 to 7298463820135882752. Current Snapshot: MvccSnapshot[applied={T|T < 7298463820282531840}]
I20260619 06:45:18.672163 30345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0eac3b29cde442f6b57bea5850c0bb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } } }
I20260619 06:45:18.672473 30345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:18.677131 30384 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0eac3b29cde442f6b57bea5850c0bb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } } }
I20260619 06:45:18.677416 30384 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:18.681257 30364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0eac3b29cde442f6b57bea5850c0bb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } } }
I20260619 06:45:18.681576 30364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:18.681883 30364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0eac3b29cde442f6b57bea5850c0bb96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eac3b29cde442f6b57bea5850c0bb96" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38477 } } peers { permanent_uuid: "b0a901906fef4c24b0652387f79bea31" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 36549 } } peers { permanent_uuid: "5fb5bf9566b94f7e8f92b74b447af8a2" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 33421 } } }
I20260619 06:45:18.682094 30364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:18.687059 30382 catalog_manager.cc:1347] Generated new cluster ID: 1aba319f95994dc19ee88a7833386d85
I20260619 06:45:18.687167 30382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:45:18.708074 30403 catalog_manager.cc:1259] Loaded cluster ID: 1aba319f95994dc19ee88a7833386d85
I20260619 06:45:18.708293 30403 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31: loading cluster ID for follower catalog manager: success
W20260619 06:45:18.709738 30403 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260619 06:45:18.709978 30403 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:45:18.753134 30382 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:45:18.755134 30382 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:45:18.773953 30382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eac3b29cde442f6b57bea5850c0bb96: Generated new TSK 0
I20260619 06:45:18.775164 30382 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:45:18.909865 30268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:18.894744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.7.146.253:36549)
0619 06:45:18.894950 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.894956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.906962 (+ 12006us) server_negotiation.cc:407] Connection header received
0619 06:45:18.908062 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.908069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.908181 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.908260 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.909587 (+ 1327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34162: BlockingRecv error: recv got EOF from 127.0.0.1:34162 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 06:45:18.912786 30343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:18.894325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.7.146.254:38477)
0619 06:45:18.894588 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.894595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.910533 (+ 15938us) server_negotiation.cc:407] Connection header received
0619 06:45:18.911011 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.911017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.911098 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.911170 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.912490 (+ 1320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40406: BlockingRecv error: recv got EOF from 127.0.0.1:40406 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 06:45:18.913331 30341 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:18.901390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.7.146.252:33421)
0619 06:45:18.901580 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.901587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.911349 (+ 9762us) server_negotiation.cc:407] Connection header received
0619 06:45:18.912032 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.912038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.912157 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.912228 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.913109 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36404: BlockingRecv error: recv got EOF from 127.0.0.1:36404 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 06:45:18.957206 30268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:18.943870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.7.146.253:36549)
0619 06:45:18.944056 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.944062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.954732 (+ 10670us) server_negotiation.cc:407] Connection header received
0619 06:45:18.955311 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.955318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.955490 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.955578 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.956817 (+ 1239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34168: BlockingRecv error: recv got EOF from 127.0.0.1:34168 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 06:45:18.996079 30268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:18.986929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.7.146.253:36549)
0619 06:45:18.987119 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:18.987125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:18.994464 (+ 7339us) server_negotiation.cc:407] Connection header received
0619 06:45:18.994662 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:18.994668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:18.994775 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:18.994919 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:18.995838 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34174: BlockingRecv error: recv got EOF from 127.0.0.1:34174 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 06:45:19.008874 30268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:19.006572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.7.146.253:36549)
0619 06:45:19.006746 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:19.006752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:19.007271 (+ 519us) server_negotiation.cc:407] Connection header received
0619 06:45:19.007493 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:19.007500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:19.007623 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:19.007736 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:19.008610 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34176: BlockingRecv error: recv got EOF from 127.0.0.1:34176 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 06:45:19.025135 30268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:19.022598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34186 (local address 127.7.146.253:36549)
0619 06:45:19.022835 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:19.022841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:19.023360 (+ 519us) server_negotiation.cc:407] Connection header received
0619 06:45:19.023582 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:19.023588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:19.023755 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:19.023877 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:19.024901 (+ 1024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34186: BlockingRecv error: recv got EOF from 127.0.0.1:34186 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260619 06:45:19.039072 30268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:19.036934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.7.146.253:36549)
0619 06:45:19.037106 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:19.037112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:19.037625 (+ 513us) server_negotiation.cc:407] Connection header received
0619 06:45:19.037804 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:19.037811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:19.037911 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:19.038024 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:19.038828 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34202: BlockingRecv error: recv got EOF from 127.0.0.1:34202 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 06:45:19.061919 30268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:19.059477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.7.146.253:36549)
0619 06:45:19.059709 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:19.059716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:19.060255 (+ 539us) server_negotiation.cc:407] Connection header received
0619 06:45:19.060507 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:19.060513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:19.060656 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:19.060790 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:19.061685 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34212: BlockingRecv error: recv got EOF from 127.0.0.1:34212 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260619 06:45:19.101610 30268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:19.098938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.7.146.253:36549)
0619 06:45:19.099131 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:19.099149 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:19.099694 (+ 545us) server_negotiation.cc:407] Connection header received
0619 06:45:19.100033 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:19.100040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:19.100151 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:19.100246 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:19.101337 (+ 1091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34228: BlockingRecv error: recv got EOF from 127.0.0.1:34228 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 06:45:19.171659 30268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:19.169784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.7.146.253:36549)
0619 06:45:19.169991 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:19.169998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:19.170340 (+ 342us) server_negotiation.cc:407] Connection header received
0619 06:45:19.170504 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:19.170510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:19.170615 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:19.170743 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:19.171429 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34232: BlockingRecv error: recv got EOF from 127.0.0.1:34232 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 06:45:19.305027 30268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:19.303129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.7.146.253:36549)
0619 06:45:19.303326 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:19.303333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:19.303662 (+ 329us) server_negotiation.cc:407] Connection header received
0619 06:45:19.303855 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:19.303859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:19.303931 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:19.304017 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:19.304843 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34244: BlockingRecv error: recv got EOF from 127.0.0.1:34244 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:45:19.404740 30356 catalog_manager.cc:1259] Loaded cluster ID: 1aba319f95994dc19ee88a7833386d85
I20260619 06:45:19.404846 30356 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2: loading cluster ID for follower catalog manager: success
I20260619 06:45:19.407389 30356 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2: acquiring CA information for follower catalog manager: success
I20260619 06:45:19.409206 30356 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5fb5bf9566b94f7e8f92b74b447af8a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:45:19.566991 30268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:19.565251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.7.146.253:36549)
0619 06:45:19.565451 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:19.565457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:19.565655 (+ 198us) server_negotiation.cc:407] Connection header received
0619 06:45:19.565822 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:19.565827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:19.565911 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:19.566056 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:19.566766 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34260: BlockingRecv error: recv got EOF from 127.0.0.1:34260 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:45:19.713404 30403 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31: acquiring CA information for follower catalog manager: success
I20260619 06:45:19.715334 30403 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b0a901906fef4c24b0652387f79bea31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:45:20.092258 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:20.086289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.7.146.253:36549)
0619 06:45:20.086788 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:20.086793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:20.086813 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:45:20.086879 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:20.086883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:20.087033 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:20.087145 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:45:20.087561 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:20.088338 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:20.089854 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:20.090283 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:20.090396 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:20.090920 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:45:20.091624 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:45:20.091772 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:20.091861 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 06:45:20.091947 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":252,"threads_started":1}
I20260619 06:45:20.103863 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:20.098354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36414 (local address 127.7.146.252:33421)
0619 06:45:20.098819 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:20.098824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:20.098844 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:45:20.099103 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:20.099108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:20.099277 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:20.099425 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:45:20.099794 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:20.100482 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:20.101776 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:20.102173 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:20.102279 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:20.102756 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:45:20.103267 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:45:20.103400 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:20.103480 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 06:45:20.103561 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":202,"threads_started":1}
I20260619 06:45:20.119029 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:20.110259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.7.146.253:36549)
0619 06:45:20.110449 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:20.110456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:20.112732 (+ 2276us) server_negotiation.cc:407] Connection header received
0619 06:45:20.112912 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:20.112918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:20.113016 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:20.113112 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:45:20.113619 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:20.114769 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:20.116993 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:20.117660 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:20.117788 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:20.118265 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:45:20.118414 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:45:20.118496 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:20.118617 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 06:45:20.118706 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:45:20.124682 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:20.111908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.7.146.252:33421)
0619 06:45:20.112091 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:20.112097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:20.119050 (+ 6953us) server_negotiation.cc:407] Connection header received
0619 06:45:20.119321 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:20.119328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:20.119425 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:20.119488 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:45:20.120078 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:20.121189 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:20.123097 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:20.123518 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:20.123625 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:20.124003 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:45:20.124105 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:45:20.124176 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:20.124311 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 06:45:20.124393 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:45:20.126199 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:20.110013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.7.146.254:38477)
0619 06:45:20.121012 (+ 10999us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:20.121018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:20.121038 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:45:20.121156 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:20.121161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:20.121384 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:20.121538 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:45:20.121982 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:20.122777 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:20.124119 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:20.124554 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:20.124702 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:20.125176 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:45:20.125625 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:45:20.125754 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:20.125868 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 06:45:20.125954 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10870,"thread_start_us":10769,"threads_started":1}
I20260619 06:45:20.131014 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 30123
I20260619 06:45:20.132139 30185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:45:20.245878 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 30123
W20260619 06:45:20.262247 30214 connection.cc:535] server connection from 127.0.0.1:34126 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:45:20.262553 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 30197
I20260619 06:45:20.263803 30259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:45:20.369392 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 30197
I20260619 06:45:20.385349 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 30271
I20260619 06:45:20.386384 30333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:45:20.492268 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 30271
2026-06-19T06:45:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4852 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:45:20 dist-test-slave-qfk4 krb5kdc[30491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:45:20 dist-test-slave-qfk4 krb5kdc[30491](info): set up 2 sockets
Jun 19 06:45:20 dist-test-slave-qfk4 krb5kdc[30491](info): commencing operation
krb5kdc: starting...
W20260619 06:45:21.513036 7755 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/test-user.keytab.
Jun 19 06:45:21 dist-test-slave-qfk4 krb5kdc[30491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:45:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.146.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
Entry for principal kudu/127.7.146.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab.
I20260619 06:45:21.757663 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38291
--webserver_interface=127.7.146.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:38291,127.7.146.253:35881,127.7.146.252:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:45:22.073338 30504 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:45:22.073655 30504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:22.073727 30504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:22.073810 30504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:22.081126 30504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:22.081228 30504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:22.081283 30504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:22.081331 30504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:22.081391 30504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:22.091246 30504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:38291,127.7.146.253:35881,127.7.146.252:45149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.254.keytab
--principal=kudu/127.7.146.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.254:38291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.146.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:22.092582 30504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:22.093822 30504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:22.105154 30509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:22.105159 30510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:22.105170 30512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:22.105818 30504 server_base.cc:1034] running on GCE node
I20260619 06:45:22.106527 30504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:22.107870 30504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:22.109068 30504 hybrid_clock.cc:648] HybridClock initialized: now 1781851522109008 us; error 33 us; skew 500 ppm
I20260619 06:45:22.109392 30504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:22 dist-test-slave-qfk4 krb5kdc[30491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851522, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:22.112355 30504 init.cc:377] Logged in from keytab as kudu/127.7.146.254@KRBTEST.COM (short username kudu)
I20260619 06:45:22.113821 30504 webserver.cc:460] Webserver started at http://127.7.146.254:33771/ using document root <none> and password file <none>
I20260619 06:45:22.114475 30504 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:22.114568 30504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:22.114861 30504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:22.117846 30504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/data/instance:
uuid: "95d9eee3847842f7beaeaa009ba09f20"
format_stamp: "Formatted at 2026-06-19 06:45:22 on dist-test-slave-qfk4"
server_key: "137aed7bbbcbfc729bd0e42cc3eb8e84"
server_key_iv: "bb0d062c34fbd9790774996efff7a304"
server_key_version: "encryptionkey@0"
I20260619 06:45:22.118573 30504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/wal/instance:
uuid: "95d9eee3847842f7beaeaa009ba09f20"
format_stamp: "Formatted at 2026-06-19 06:45:22 on dist-test-slave-qfk4"
server_key: "137aed7bbbcbfc729bd0e42cc3eb8e84"
server_key_iv: "bb0d062c34fbd9790774996efff7a304"
server_key_version: "encryptionkey@0"
I20260619 06:45:22.123615 30504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:45:22.126991 30519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:22.128317 30504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:45:22.128471 30504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/wal
uuid: "95d9eee3847842f7beaeaa009ba09f20"
format_stamp: "Formatted at 2026-06-19 06:45:22 on dist-test-slave-qfk4"
server_key: "137aed7bbbcbfc729bd0e42cc3eb8e84"
server_key_iv: "bb0d062c34fbd9790774996efff7a304"
server_key_version: "encryptionkey@0"
I20260619 06:45:22.128713 30504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:22.145566 30504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:22.149631 30504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:22.149891 30504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:22.167407 30504 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.254:38291
I20260619 06:45:22.167416 30570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.254:38291 every 8 connection(s)
I20260619 06:45:22.169133 30504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/data/info.pb
I20260619 06:45:22.170971 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 30504
I20260619 06:45:22.171247 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-0/wal/instance
I20260619 06:45:22.171991 7755 external_mini_cluster.cc:1428] Setting key 3950c75191e1d658b1face06e9c1a4ae
I20260619 06:45:22.173834 30571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:22.181327 30571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.146.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.253@KRBTEST.COM" created.
Jun 19 06:45:22 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851522, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:45:22.206408 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.184728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38291 (local address 127.0.0.1:36448)
0619 06:45:22.185962 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.185996 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:22.187827 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:22.188832 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:22.188839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:22.189620 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:22.190088 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:22.190112 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.191792 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.191798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:22.193598 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:22.193615 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.195047 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.195058 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:22.195299 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:22.196675 (+ 1376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:22.196699 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:22.199079 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:22.202369 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:22.202381 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:22.202394 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:22.202892 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:22.203300 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:22.203304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:22.203307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:22.203494 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:22.204205 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:22.204213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:22.204466 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 06:45:22.204753 (+ 287us) client_negotiation.cc:240] Negotiation successful
0619 06:45:22.205384 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":107,"thread_start_us":842,"threads_started":1}
I20260619 06:45:22.206442 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.184728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.7.146.254:38291)
0619 06:45:22.185536 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:22.185547 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:22.186130 (+ 583us) server_negotiation.cc:407] Connection header received
0619 06:45:22.188211 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:22.188218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:22.188527 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:22.188774 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:22.190320 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.191608 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.193778 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.194674 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.194861 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:22.199336 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:22.199343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:22.199357 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:22.199387 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:22.202229 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:22.203022 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:22.203029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:22.203031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:22.203188 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:22.203588 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:22.203591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:22.203593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:22.204077 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:22.204234 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:22.204933 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 06:45:22.205209 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":361,"threads_started":1}
I20260619 06:45:22.209345 30571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:45:22.210515 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.210083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41130)
0619 06:45:22.210266 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.210388 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 06:45:22.210775 30523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
I20260619 06:45:22.212284 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.211894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41138)
0619 06:45:22.212046 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.212126 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:45:22.212460 30571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:35881: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab.
Entry for principal kudu/127.7.146.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab.
I20260619 06:45:22.227779 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.253:35881
--webserver_interface=127.7.146.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab
--principal=kudu/127.7.146.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:38291,127.7.146.253:35881,127.7.146.252:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:45:22.261824 30571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } attempt: 1
I20260619 06:45:22.263191 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.262625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41150)
0619 06:45:22.262849 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.262995 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 06:45:22.264182 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.263779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41156)
0619 06:45:22.263931 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.264020 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:45:22.264359 30571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:35881: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
I20260619 06:45:22.332617 30571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } attempt: 2
I20260619 06:45:22.333861 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.333252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41158)
0619 06:45:22.333549 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.333673 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 06:45:22.334914 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.334495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41162)
0619 06:45:22.334673 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.334759 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 06:45:22.335140 30571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:35881: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
I20260619 06:45:22.426438 30571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } attempt: 3
I20260619 06:45:22.427681 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.427157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41178)
0619 06:45:22.427397 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.427510 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 06:45:22.428798 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.428294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41192)
0619 06:45:22.428470 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.428610 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 06:45:22.429039 30571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:35881: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
W20260619 06:45:22.547029 30578 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:45:22.547349 30578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:22.547418 30578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:22.547498 30578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:22.554750 30578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:22.554847 30578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:22.554903 30578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:22.554950 30578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:22.555009 30578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:22.564834 30578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:38291,127.7.146.253:35881,127.7.146.252:45149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.253.keytab
--principal=kudu/127.7.146.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.253:35881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.146.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:22.566102 30578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:22.567381 30578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:45:22.572319 30571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } attempt: 4
I20260619 06:45:22.573746 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.573068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41204)
0619 06:45:22.573440 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.573556 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260619 06:45:22.574887 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.574451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41210)
0619 06:45:22.574639 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.574735 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260619 06:45:22.575084 30571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.253:35881: Network error: Client connection negotiation failed: client connection to 127.7.146.253:35881: connect: Connection refused (error 111)
W20260619 06:45:22.579401 30583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:22.579358 30586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:22.579424 30584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:22.580386 30578 server_base.cc:1034] running on GCE node
I20260619 06:45:22.581207 30578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:22.582588 30578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:22.583812 30578 hybrid_clock.cc:648] HybridClock initialized: now 1781851522583741 us; error 45 us; skew 500 ppm
I20260619 06:45:22.584131 30578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:22 dist-test-slave-qfk4 krb5kdc[30491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851522, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:22.587257 30578 init.cc:377] Logged in from keytab as kudu/127.7.146.253@KRBTEST.COM (short username kudu)
I20260619 06:45:22.588694 30578 webserver.cc:460] Webserver started at http://127.7.146.253:36567/ using document root <none> and password file <none>
I20260619 06:45:22.589387 30578 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:22.589488 30578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:22.589807 30578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:22.592806 30578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/data/instance:
uuid: "1c705b4aac1b41d595300bbf65aa2e20"
format_stamp: "Formatted at 2026-06-19 06:45:22 on dist-test-slave-qfk4"
server_key: "aa1215a0d147aad13773f82a71603289"
server_key_iv: "3427a0a0607722eeeb900d78e40eb64b"
server_key_version: "encryptionkey@0"
I20260619 06:45:22.593569 30578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/wal/instance:
uuid: "1c705b4aac1b41d595300bbf65aa2e20"
format_stamp: "Formatted at 2026-06-19 06:45:22 on dist-test-slave-qfk4"
server_key: "aa1215a0d147aad13773f82a71603289"
server_key_iv: "3427a0a0607722eeeb900d78e40eb64b"
server_key_version: "encryptionkey@0"
I20260619 06:45:22.598747 30578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:45:22.602094 30593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:22.603456 30578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:45:22.603611 30578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/wal
uuid: "1c705b4aac1b41d595300bbf65aa2e20"
format_stamp: "Formatted at 2026-06-19 06:45:22 on dist-test-slave-qfk4"
server_key: "aa1215a0d147aad13773f82a71603289"
server_key_iv: "3427a0a0607722eeeb900d78e40eb64b"
server_key_version: "encryptionkey@0"
I20260619 06:45:22.603804 30578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:22.627178 30578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:22.631453 30578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:22.631692 30578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:22.648370 30578 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.253:35881
I20260619 06:45:22.648379 30644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.253:35881 every 8 connection(s)
I20260619 06:45:22.649677 30578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/data/info.pb
I20260619 06:45:22.650818 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 30578
I20260619 06:45:22.650992 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-1/wal/instance
I20260619 06:45:22.651506 7755 external_mini_cluster.cc:1428] Setting key 80383f8afb6d80fb1d59d2005b4a18a3
I20260619 06:45:22.654070 30645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:22.662410 30645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:45:22 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851522, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.146.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.146.252@KRBTEST.COM" created.
I20260619 06:45:22.682029 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.665033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.7.146.254:38291)
0619 06:45:22.665192 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:22.665196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:22.666621 (+ 1425us) server_negotiation.cc:407] Connection header received
0619 06:45:22.668490 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:22.668497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:22.668613 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:22.668705 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:22.670217 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.670904 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.672413 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.672869 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.672981 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:22.676312 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:22.676317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:22.676320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:22.676355 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:22.678915 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:22.679629 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:22.679634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:22.679637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:22.679785 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:22.680332 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:22.680337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:22.680341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:22.680711 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:22.680853 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:22.681526 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 06:45:22.681746 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:45:22.682961 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.665725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38291 (local address 127.0.0.1:36464)
0619 06:45:22.666453 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.666497 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:22.668201 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:22.668954 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:22.668962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:22.669487 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:22.670041 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:22.670058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.671054 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.671057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:22.672266 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:22.672277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.673077 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.673085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:22.673304 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:22.673974 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:22.673995 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:22.676052 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:22.679058 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:22.679065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:22.679078 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:22.679510 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:22.679983 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:22.679989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:22.679992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:22.680217 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:22.680849 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:22.680856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:22.681196 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 06:45:22.681420 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 06:45:22.681827 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":293,"threads_started":1}
I20260619 06:45:22.684646 30645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:45:22 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851522, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
I20260619 06:45:22.699807 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.685427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41222)
0619 06:45:22.685609 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.685628 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:22.685730 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:22.686674 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:22.686679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:22.687095 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:22.687244 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:22.687253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.689162 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.689166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:22.689979 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:22.689989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.691027 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.691031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:22.691177 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:22.691747 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:22.691765 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:22.693714 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:22.697376 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:22.697381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:22.697383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:22.697780 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:22.698201 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:22.698205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:22.698207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:22.698369 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:22.699024 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:22.699027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:22.699142 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 06:45:22.699309 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 06:45:22.699536 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:45:22.700160 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.685594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.7.146.253:35881)
0619 06:45:22.686263 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:22.686269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:22.686294 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:45:22.686336 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:22.686341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:22.686524 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:22.686633 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:22.687411 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.689038 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.690107 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.690891 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.691006 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:22.693934 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:22.693948 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:22.693967 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:22.694004 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:22.697205 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:22.697916 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:22.697925 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:22.697929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:22.698073 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:22.698472 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:22.698477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:22.698479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:22.698912 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:22.699117 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:22.699475 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 06:45:22.699739 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":376,"threads_started":1}
I20260619 06:45:22.702639 30645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:45:22.703667 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.703208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37444)
0619 06:45:22.703396 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.703521 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 06:45:22.703998 30597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
I20260619 06:45:22.705582 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.705255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37446)
0619 06:45:22.705397 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.705477 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:45:22.705806 30645 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:45149: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.146.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab.
Entry for principal kudu/127.7.146.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab.
I20260619 06:45:22.711561 7755 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
/tmp/dist-test-taskjzr1uN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.252:45149
--webserver_interface=127.7.146.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab
--principal=kudu/127.7.146.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.146.212:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.146.254:38291,127.7.146.253:35881,127.7.146.252:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:45:22.755123 30645 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } attempt: 1
I20260619 06:45:22.756428 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.755872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37462)
0619 06:45:22.756100 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.756238 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 06:45:22.757561 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.757133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37470)
0619 06:45:22.757305 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.757405 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 06:45:22.757807 30645 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:45149: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
I20260619 06:45:22.826056 30645 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } attempt: 2
I20260619 06:45:22.827286 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.826768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37486)
0619 06:45:22.826985 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.827099 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:45:22.828302 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.827935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37490)
0619 06:45:22.828081 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.828166 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:45:22.828502 30645 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:45149: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
I20260619 06:45:22.874379 30571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } attempt: 5
Jun 19 06:45:22 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851522, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
I20260619 06:45:22.887604 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.875218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.7.146.253:35881)
0619 06:45:22.875392 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:22.875398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:22.875495 (+ 97us) server_negotiation.cc:407] Connection header received
0619 06:45:22.875675 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:22.875680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:22.875776 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:22.875863 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:22.876697 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.877392 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.878553 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.878999 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.879117 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:22.882513 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:22.882518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:22.882522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:22.882556 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:22.885075 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:22.885884 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:22.885888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:22.885890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:22.886006 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:22.886399 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:22.886402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:22.886403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:22.886670 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:22.886794 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:22.887208 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 06:45:22.887375 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:45:22.887800 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.875158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41234)
0619 06:45:22.875391 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.875410 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:22.875551 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:22.875922 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:22.875926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:22.876332 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:22.876516 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:22.876526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.877527 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.877532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:22.878402 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:22.878415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:22.879150 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:22.879156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:22.879385 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:22.879978 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:22.879996 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:22.882304 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:22.885216 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:22.885222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:22.885226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:22.885777 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:22.886109 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:22.886114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:22.886117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:22.886298 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:22.886785 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:22.886790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:22.886987 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 06:45:22.887142 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 06:45:22.887440 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 06:45:22.888859 30571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:45:22.889952 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.889440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37500)
0619 06:45:22.889650 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.889763 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 06:45:22.890918 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.890516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37502)
0619 06:45:22.890677 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.890789 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:45:22.891155 30571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:45149: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
I20260619 06:45:22.919798 30645 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } attempt: 3
I20260619 06:45:22.921118 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.920493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37504)
0619 06:45:22.920759 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.920889 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 06:45:22.922260 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.921698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37510)
0619 06:45:22.921965 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.922087 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260619 06:45:22.922487 30645 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:45149: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
I20260619 06:45:22.942386 30571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } attempt: 1
I20260619 06:45:22.943579 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.943064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37512)
0619 06:45:22.943275 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.943389 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 06:45:22.944535 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:22.944180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37520)
0619 06:45:22.944317 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:22.944404 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:45:22.944828 30571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:45149: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
I20260619 06:45:23.013067 30571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } attempt: 2
I20260619 06:45:23.014215 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.013720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37528)
0619 06:45:23.013951 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:23.014043 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 06:45:23.015254 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.014916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37540)
0619 06:45:23.015053 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:23.015129 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 06:45:23.015475 30571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:45149: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
W20260619 06:45:23.032694 30652 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:45:23.033067 30652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:45:23.033167 30652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:45:23.033281 30652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:45:23.040482 30652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:45:23.040611 30652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:45:23.040704 30652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:45:23.040766 30652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:45:23.040867 30652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:45:23.050757 30652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.146.212:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.146.254:38291,127.7.146.253:35881,127.7.146.252:45149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/krb5kdc/kudu_127.7.146.252.keytab
--principal=kudu/127.7.146.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.146.252:45149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.146.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:45:23.052053 30652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:45:23.053413 30652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:45:23.064800 30657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:23.064888 30658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:45:23.064844 30660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:45:23.065412 30652 server_base.cc:1034] running on GCE node
I20260619 06:45:23.065675 30645 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } attempt: 4
I20260619 06:45:23.066293 30652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:45:23.066756 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.066235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37552)
0619 06:45:23.066470 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:23.066571 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 06:45:23.067685 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.067382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37566)
0619 06:45:23.067496 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:23.067563 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 06:45:23.067885 30645 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:45149: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
I20260619 06:45:23.068133 30652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:45:23.069356 30652 hybrid_clock.cc:648] HybridClock initialized: now 1781851523069279 us; error 39 us; skew 500 ppm
I20260619 06:45:23.069707 30652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:45:23 dist-test-slave-qfk4 krb5kdc[30491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:45:23.072777 30652 init.cc:377] Logged in from keytab as kudu/127.7.146.252@KRBTEST.COM (short username kudu)
I20260619 06:45:23.074239 30652 webserver.cc:460] Webserver started at http://127.7.146.252:35543/ using document root <none> and password file <none>
I20260619 06:45:23.074913 30652 fs_manager.cc:359] Metadata directory not provided
I20260619 06:45:23.075008 30652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:45:23.075301 30652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:45:23.078184 30652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/data/instance:
uuid: "f53b362622e74c929406229880751a86"
format_stamp: "Formatted at 2026-06-19 06:45:23 on dist-test-slave-qfk4"
server_key: "33646eb7c43a6fdf9039ad39e591b600"
server_key_iv: "f676b0138be9a1e0f0e90eceb0ad0a24"
server_key_version: "encryptionkey@0"
I20260619 06:45:23.078912 30652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/wal/instance:
uuid: "f53b362622e74c929406229880751a86"
format_stamp: "Formatted at 2026-06-19 06:45:23 on dist-test-slave-qfk4"
server_key: "33646eb7c43a6fdf9039ad39e591b600"
server_key_iv: "f676b0138be9a1e0f0e90eceb0ad0a24"
server_key_version: "encryptionkey@0"
I20260619 06:45:23.084038 30652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260619 06:45:23.087404 30667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:45:23.088770 30652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:45:23.088922 30652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/data,/tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/wal
uuid: "f53b362622e74c929406229880751a86"
format_stamp: "Formatted at 2026-06-19 06:45:23 on dist-test-slave-qfk4"
server_key: "33646eb7c43a6fdf9039ad39e591b600"
server_key_iv: "f676b0138be9a1e0f0e90eceb0ad0a24"
server_key_version: "encryptionkey@0"
I20260619 06:45:23.089114 30652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:45:23.112375 30652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:45:23.116470 30652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:45:23.116752 30652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:45:23.121654 30571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } attempt: 3
I20260619 06:45:23.122795 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.122212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37574)
0619 06:45:23.122484 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:23.122600 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 06:45:23.123880 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.123488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37588)
0619 06:45:23.123663 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:23.123747 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 06:45:23.124166 30571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.146.252:45149: Network error: Client connection negotiation failed: client connection to 127.7.146.252:45149: connect: Connection refused (error 111)
I20260619 06:45:23.133766 30652 rpc_server.cc:307] RPC server started. Bound to: 127.7.146.252:45149
I20260619 06:45:23.133766 30718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.146.252:45149 every 8 connection(s)
I20260619 06:45:23.135145 30652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/data/info.pb
I20260619 06:45:23.138707 30719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:45:23.144771 7755 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu as pid 30652
I20260619 06:45:23.144942 7755 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjzr1uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851174422545-7755-0/minicluster-data/master-2/wal/instance
I20260619 06:45:23.145367 7755 external_mini_cluster.cc:1428] Setting key 194e449dee1045f5ba138713cfbb9c2a
I20260619 06:45:23.145071 30719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:45:23 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
Jun 19 06:45:23 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
I20260619 06:45:23.165405 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.147645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.7.146.254:38291)
0619 06:45:23.147835 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.147840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.148431 (+ 591us) server_negotiation.cc:407] Connection header received
0619 06:45:23.149987 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.149993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.150066 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.150162 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.151451 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.152166 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.153888 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.154474 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.154562 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.158255 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:23.158261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:23.158264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:23.158299 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:23.161704 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.162519 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.162527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.162531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.162753 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.163317 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.163321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.163323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.163665 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:23.163840 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:23.164759 (+ 919us) server_negotiation.cc:299] Negotiation successful
0619 06:45:23.164994 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:45:23.166682 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.147696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.254:38291 (local address 127.0.0.1:36476)
0619 06:45:23.148290 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:23.148332 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:23.149743 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:23.150254 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:23.150261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:23.150937 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:23.151287 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:23.151303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.152340 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.152345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:23.153751 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:23.153763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.154685 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.154693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:23.154915 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.155684 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:23.155706 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:23.157969 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:23.161839 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:23.161850 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:23.161860 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:23.162375 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:23.162887 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:23.162892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:23.162896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:23.163140 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:23.163857 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:23.163866 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:23.164349 (+ 483us) client_negotiation.cc:769] Sending connection context
0619 06:45:23.164627 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 06:45:23.165407 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":283,"threads_started":1}
I20260619 06:45:23.166724 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.147118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.7.146.252:45149)
0619 06:45:23.147911 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.147929 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.147981 (+ 52us) server_negotiation.cc:407] Connection header received
0619 06:45:23.149736 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.149758 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.149992 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.150157 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.151528 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.152984 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.154340 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.155018 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.155151 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.159125 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:23.159140 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:23.159158 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:23.159208 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:23.162640 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.163547 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.163556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.163559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.163752 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.164153 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.164158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.164161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.164515 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:23.164875 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:23.165205 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 06:45:23.165564 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":574,"thread_start_us":369,"threads_started":1}
I20260619 06:45:23.168252 30719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:45:23 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
Jun 19 06:45:23 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.254@KRBTEST.COM
I20260619 06:45:23.182415 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.169239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.7.146.253:35881)
0619 06:45:23.169411 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.169416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.169434 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:45:23.169489 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.169494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.169564 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.169666 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.170447 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.171124 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.172303 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.172975 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.173104 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.176326 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:23.176333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:23.176338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:23.176380 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:23.179710 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.180386 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.180391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.180395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.180552 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.181088 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.181092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.181095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.181384 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:23.181516 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:23.181958 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 06:45:23.182142 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:45:23.182497 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.168959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.253:35881 (local address 127.0.0.1:41246)
0619 06:45:23.169241 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:23.169260 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:23.169352 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:23.169702 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:23.169707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:23.170074 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:23.170263 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:23.170274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.171273 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.171276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:23.172149 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:23.172159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.173127 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.173132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:23.173334 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.174103 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:23.174125 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:23.176133 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:23.179861 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:23.179865 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:23.179868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:23.180271 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:23.180720 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:23.180723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:23.180759 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:23.180957 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:23.181525 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:23.181531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:23.181696 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 06:45:23.181852 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 06:45:23.182139 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260619 06:45:23.183393 30719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:45:23 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851523, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.252@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
I20260619 06:45:23.192842 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.170375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.7.146.254:38291)
0619 06:45:23.172731 (+ 2356us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.172735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.172749 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:45:23.172790 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.172793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.172847 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.172929 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.174531 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.175662 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.177398 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.177836 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.177932 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.186847 (+ 8915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:23.186855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:23.186860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:23.186917 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:23.190008 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.190823 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.190828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.190831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.191000 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.191377 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.191383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.191386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.191734 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:23.192011 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:23.192252 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 06:45:23.192468 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2247}
I20260619 06:45:23.196610 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.184062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37612)
0619 06:45:23.184236 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:23.184251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:23.184336 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:23.184656 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:23.184660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:23.185025 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:23.185148 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:23.185164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.186030 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.186035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:23.186953 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:23.186966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.187668 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.187673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:23.187907 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.188379 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:23.188402 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:23.190716 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:23.194066 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:23.194070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:23.194073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:23.194504 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:23.194900 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:23.194905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:23.194909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:23.195050 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:23.195579 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:23.195584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:23.195749 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 06:45:23.195918 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 06:45:23.196219 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:45:23.196666 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.184279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.7.146.252:45149)
0619 06:45:23.184422 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.184428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.184442 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:45:23.184473 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.184475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.184522 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.184618 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.185277 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.185919 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.187082 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.187520 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.187644 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.190897 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:23.190903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:23.190908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:23.190943 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:23.193951 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.194629 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.194633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.194636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.194779 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.195160 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.195165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.195168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.195464 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:23.195567 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:23.196006 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 06:45:23.196349 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":48}
I20260619 06:45:23.202834 30719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86: Bootstrap starting.
Jun 19 06:45:23 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.146.253@KRBTEST.COM
I20260619 06:45:23.206508 30719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:23.207624 30719 log.cc:824] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:23.210415 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.194674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.7.146.253:35881)
0619 06:45:23.194881 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.194887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.194904 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:45:23.194955 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.194958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.195015 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.195081 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.196508 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.197798 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.199476 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.200022 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.200121 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.204958 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:23.204963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:23.204967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:23.205001 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:23.207744 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.208630 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.208636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.208639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.208812 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.209199 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.209203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.209205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.209463 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:23.209661 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:23.209964 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 06:45:23.210134 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:45:23.210763 30719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86: No bootstrap required, opened a new log
I20260619 06:45:23.214869 7755 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:45:23.218698 30719 raft_consensus.cc:348] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } }
I20260619 06:45:23.219110 30719 raft_consensus.cc:729] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53b362622e74c929406229880751a86, State: Initialized, Role: FOLLOWER
I20260619 06:45:23.219630 30719 consensus_queue.cc:260] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } }
I20260619 06:45:23.220498 30725 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } } }
I20260619 06:45:23.217149 30546 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "95d9eee3847842f7beaeaa009ba09f20"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36488
I20260619 06:45:23.220871 30725 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:23.221848 30719 sys_catalog.cc:564] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:45:23.230545 30736 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:45:23.230676 30736 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:45:23.228543 30652 master_runner.cc:186] Error getting master registration for 127.7.146.254:38291: OK, instance_id { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" instance_seqno: 1781851522162048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:45:23.239105 30652 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:45:23.301378 30571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } attempt: 4
Jun 19 06:45:23 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851522, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.254@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
I20260619 06:45:23.318709 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.302300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.7.146.252:45149)
0619 06:45:23.302495 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.302502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.302524 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:45:23.302609 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.302615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.302704 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.302816 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.304025 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.305207 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.307154 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.307912 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.308043 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.312098 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:23.312106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:23.312111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:23.312154 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:23.315720 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.316388 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.316393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.316397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.316559 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.317011 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.317016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.317019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.317494 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:23.317637 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:23.318096 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 06:45:23.318330 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:45:23.318758 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.302133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37614)
0619 06:45:23.302343 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:23.302364 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:23.302478 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:23.303148 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:23.303153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:23.303631 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:23.303839 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:23.303850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.305391 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.305398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:23.306987 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:23.307006 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.308101 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.308108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:23.308335 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.309097 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:23.309129 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:23.311868 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:23.315857 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:23.315861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:23.315864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:23.316272 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:23.316704 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:23.316709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:23.316713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:23.316897 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:23.317617 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:23.317623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:23.317837 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 06:45:23.317998 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 06:45:23.318305 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 06:45:23.325067 30571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20: Bootstrap starting.
I20260619 06:45:23.328634 30571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20: Neither blocks nor log segments found. Creating new log.
I20260619 06:45:23.329735 30571 log.cc:824] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:23.332713 30571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20: No bootstrap required, opened a new log
I20260619 06:45:23.338141 30571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } }
I20260619 06:45:23.338479 30571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d9eee3847842f7beaeaa009ba09f20, State: Initialized, Role: FOLLOWER
I20260619 06:45:23.338927 30571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } }
I20260619 06:45:23.339859 30739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } } }
I20260619 06:45:23.340269 30739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:23.341164 30571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:23.341320 30546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 06:45:23.341559 30546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:23.345105 30546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } }
I20260619 06:45:23.346745 30546 leader_election.cc:288] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [CANDIDATE]: Term 1 election: Requested vote from peers 1c705b4aac1b41d595300bbf65aa2e20 (127.7.146.253:35881), f53b362622e74c929406229880751a86 (127.7.146.252:45149)
I20260619 06:45:23.347863 30694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95d9eee3847842f7beaeaa009ba09f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f53b362622e74c929406229880751a86"
I20260619 06:45:23.348417 30694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:23.352556 30694 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d9eee3847842f7beaeaa009ba09f20 in term 1.
I20260619 06:45:23.353300 30523 leader_election.cc:302] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d9eee3847842f7beaeaa009ba09f20, f53b362622e74c929406229880751a86; no voters:
I20260619 06:45:23.353776 30739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:45:23.360890 30739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [term 1 LEADER]: Becoming Leader. State: Replica: 95d9eee3847842f7beaeaa009ba09f20, State: Running, Role: LEADER
I20260619 06:45:23.361897 30739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } }
I20260619 06:45:23.367048 30645 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } attempt: 5
W20260619 06:45:23.368746 30673 tablet.cc:2369] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:45:23 dist-test-slave-qfk4 krb5kdc[30491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851522, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.146.253@KRBTEST.COM for kudu/127.7.146.252@KRBTEST.COM
I20260619 06:45:23.385133 30750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d9eee3847842f7beaeaa009ba09f20. Latest consensus state: current_term: 1 leader_uuid: "95d9eee3847842f7beaeaa009ba09f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } } }
I20260619 06:45:23.385630 30750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:23.388233 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.368429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.7.146.252:45149)
0619 06:45:23.369583 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.369588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.369608 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:45:23.369667 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.369673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.369778 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.369887 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.371035 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.372134 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.373977 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.374632 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.374757 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.379681 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:23.379688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:23.379693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:23.379743 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:23.383957 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.385234 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.385240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.385255 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.385416 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.386286 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.386292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.386296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.386662 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:23.386813 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:23.387591 (+ 778us) server_negotiation.cc:299] Negotiation successful
0619 06:45:23.387862 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019}
I20260619 06:45:23.389950 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.367675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.146.252:45149 (local address 127.0.0.1:37620)
0619 06:45:23.368087 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 06:45:23.368107 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:45:23.368228 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:45:23.370151 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:45:23.370157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:45:23.370644 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:45:23.370856 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:23.370869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.372333 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.372339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:23.373780 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:45:23.373795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.375305 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.375312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:45:23.375535 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.376210 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:45:23.376233 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:45:23.379414 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:45:23.384167 (+ 4753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:23.384172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:23.384176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:23.385049 (+ 873us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:23.385805 (+ 756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:45:23.385811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:45:23.385814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:45:23.386090 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:45:23.386938 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:45:23.386944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:45:23.387203 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 06:45:23.388854 (+ 1651us) client_negotiation.cc:240] Negotiation successful
0619 06:45:23.389366 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280}
I20260619 06:45:23.397060 30645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20: Bootstrap starting.
I20260619 06:45:23.401723 30757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:45:23.401973 30645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20: Neither blocks nor log segments found. Creating new log.
W20260619 06:45:23.402606 30525 tablet.cc:2369] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:45:23.403321 30645 log.cc:824] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20: Log is configured to *not* fsync() on all Append() calls
I20260619 06:45:23.416698 30757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:45:23.418079 30645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20: No bootstrap required, opened a new log
I20260619 06:45:23.439043 30694 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [term 1 FOLLOWER]: Refusing update from remote peer 95d9eee3847842f7beaeaa009ba09f20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:45:23.439589 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.380702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.7.146.254:38291)
0619 06:45:23.380937 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.380943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.391492 (+ 10549us) server_negotiation.cc:407] Connection header received
0619 06:45:23.391911 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.391918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.392002 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.392070 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.398460 (+ 6390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.399594 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.407132 (+ 7538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.407833 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.407930 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.419687 (+ 11757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:23.419695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:23.419699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:23.419766 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:23.423599 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.434038 (+ 10439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.434045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.434048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.434247 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.435511 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.435517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.435520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.435975 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:23.436239 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:23.438731 (+ 2492us) server_negotiation.cc:299] Negotiation successful
0619 06:45:23.439030 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 06:45:23.440187 30750 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:45:23.442325 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.394768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.7.146.252:45149)
0619 06:45:23.394967 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.394973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.397392 (+ 2419us) server_negotiation.cc:407] Connection header received
0619 06:45:23.397532 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.397537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.397631 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.397733 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.406390 (+ 8657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.407575 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.411390 (+ 3815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.412075 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.412188 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.424508 (+ 12320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:23.424516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:23.424522 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:23.424592 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:23.428325 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.434863 (+ 6538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.434870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.434873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.435034 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.437107 (+ 2073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.437113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.437117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.437526 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:23.437798 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:23.441599 (+ 3801us) server_negotiation.cc:299] Negotiation successful
0619 06:45:23.441917 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 06:45:23.445819 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:23.381514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.7.146.253:35881)
0619 06:45:23.381691 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.381696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.394608 (+ 12912us) server_negotiation.cc:407] Connection header received
0619 06:45:23.395176 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.395182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.395260 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.395328 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.403518 (+ 8190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.404691 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.411925 (+ 7234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:23.412533 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:23.412684 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:23.429500 (+ 16816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:45:23.429508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:45:23.429513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:45:23.429569 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:45:23.433050 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.436755 (+ 3705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.436762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.436766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.436948 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:45:23.440683 (+ 3735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:45:23.440689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:45:23.440693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:45:23.441107 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:45:23.441352 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:23.445097 (+ 3745us) server_negotiation.cc:299] Negotiation successful
0619 06:45:23.445355 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:45:23.447654 30763 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:45:23.451889 30504 master_runner.cc:186] Error getting master registration for 127.7.146.253:35881: OK, instance_id { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" instance_seqno: 1781851522643703 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:45:23.470805 30504 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:45:23.467476 30645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } }
I20260619 06:45:23.471369 30645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c705b4aac1b41d595300bbf65aa2e20, State: Initialized, Role: FOLLOWER
I20260619 06:45:23.471949 30645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } }
I20260619 06:45:23.473371 30765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } } }
I20260619 06:45:23.474186 30765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:23.477566 30645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:45:23.477653 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95d9eee3847842f7beaeaa009ba09f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1c705b4aac1b41d595300bbf65aa2e20"
I20260619 06:45:23.478130 30620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:45:23.483557 30725 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d9eee3847842f7beaeaa009ba09f20. Latest consensus state: current_term: 1 leader_uuid: "95d9eee3847842f7beaeaa009ba09f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } } }
I20260619 06:45:23.483872 30725 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:23.496920 30620 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d9eee3847842f7beaeaa009ba09f20 in term 1.
I20260619 06:45:23.498024 30619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [term 1 FOLLOWER]: Refusing update from remote peer 95d9eee3847842f7beaeaa009ba09f20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:45:23.501945 30750 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:45:23.505813 30725 sys_catalog.cc:455] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95d9eee3847842f7beaeaa009ba09f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } } }
I20260619 06:45:23.506083 30725 sys_catalog.cc:458] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:23.531608 30750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95d9eee3847842f7beaeaa009ba09f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } } }
I20260619 06:45:23.531899 30750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:23.534256 30757 catalog_manager.cc:1347] Generated new cluster ID: be30b3de4199441aa13e99b7dd11a0b8
I20260619 06:45:23.534363 30757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:45:23.564093 30757 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:45:23.566102 30757 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:45:23.577275 30765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d9eee3847842f7beaeaa009ba09f20. Latest consensus state: current_term: 1 leader_uuid: "95d9eee3847842f7beaeaa009ba09f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } } }
I20260619 06:45:23.577589 30765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:23.577803 30778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95d9eee3847842f7beaeaa009ba09f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } } }
I20260619 06:45:23.578194 30778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20 [sys.catalog]: This master's current role is: LEADER
I20260619 06:45:23.578088 30776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95d9eee3847842f7beaeaa009ba09f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9eee3847842f7beaeaa009ba09f20" member_type: VOTER last_known_addr { host: "127.7.146.254" port: 38291 } } peers { permanent_uuid: "1c705b4aac1b41d595300bbf65aa2e20" member_type: VOTER last_known_addr { host: "127.7.146.253" port: 35881 } } peers { permanent_uuid: "f53b362622e74c929406229880751a86" member_type: VOTER last_known_addr { host: "127.7.146.252" port: 45149 } } }
I20260619 06:45:23.578588 30776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:45:23.597817 30757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95d9eee3847842f7beaeaa009ba09f20: Generated new TSK 0
I20260619 06:45:23.599015 30757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:45:23.604214 30784 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 06:45:23.696087 30574 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:23.686546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.7.146.254:38291)
0619 06:45:23.686771 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.686777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.693555 (+ 6778us) server_negotiation.cc:407] Connection header received
0619 06:45:23.694561 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.694568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.694665 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.694739 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.695839 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36498: BlockingRecv error: recv got EOF from 127.0.0.1:36498 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 06:45:23.699738 30649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:23.686139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.7.146.253:35881)
0619 06:45:23.686331 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.686338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.697592 (+ 11254us) server_negotiation.cc:407] Connection header received
0619 06:45:23.698296 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.698302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.698398 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.698469 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.699480 (+ 1011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41284: BlockingRecv error: recv got EOF from 127.0.0.1:41284 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 06:45:23.703233 30722 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:23.694206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.7.146.252:45149)
0619 06:45:23.694426 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.694432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.701161 (+ 6729us) server_negotiation.cc:407] Connection header received
0619 06:45:23.701897 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.701903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.701985 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.702057 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.702981 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37634: BlockingRecv error: recv got EOF from 127.0.0.1:37634 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260619 06:45:23.710348 30649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:23.708101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.7.146.253:35881)
0619 06:45:23.708276 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.708282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.708776 (+ 494us) server_negotiation.cc:407] Connection header received
0619 06:45:23.708976 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.708983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.709125 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.709238 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.710097 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41286: BlockingRecv error: recv got EOF from 127.0.0.1:41286 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 06:45:23.723912 30649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:23.718556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41292 (local address 127.7.146.253:35881)
0619 06:45:23.718728 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.718734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.722356 (+ 3622us) server_negotiation.cc:407] Connection header received
0619 06:45:23.722557 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.722563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.722665 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.722772 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.723703 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41292: BlockingRecv error: recv got EOF from 127.0.0.1:41292 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 06:45:23.731091 30649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:23.728975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.7.146.253:35881)
0619 06:45:23.729152 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.729158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.729639 (+ 481us) server_negotiation.cc:407] Connection header received
0619 06:45:23.729847 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.729852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.729943 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.730032 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.730868 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41304: BlockingRecv error: recv got EOF from 127.0.0.1:41304 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 06:45:23.744481 30649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:23.742137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.7.146.253:35881)
0619 06:45:23.742372 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.742377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.742940 (+ 563us) server_negotiation.cc:407] Connection header received
0619 06:45:23.743147 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.743152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.743246 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.743339 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.744264 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41312: BlockingRecv error: recv got EOF from 127.0.0.1:41312 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260619 06:45:23.762347 30649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:23.760086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.7.146.253:35881)
0619 06:45:23.760285 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.760292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.760877 (+ 585us) server_negotiation.cc:407] Connection header received
0619 06:45:23.761075 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.761081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.761172 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.761273 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.762119 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41320: BlockingRecv error: recv got EOF from 127.0.0.1:41320 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 06:45:23.784271 30649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:23.782139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.7.146.253:35881)
0619 06:45:23.782379 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.782385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.782744 (+ 359us) server_negotiation.cc:407] Connection header received
0619 06:45:23.782956 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.782960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.783033 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.783118 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.784004 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41332: BlockingRecv error: recv got EOF from 127.0.0.1:41332 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260619 06:45:23.821866 30649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:23.819902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.7.146.253:35881)
0619 06:45:23.820107 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.820114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.820458 (+ 344us) server_negotiation.cc:407] Connection header received
0619 06:45:23.820664 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.820671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.820770 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.820909 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.821602 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41346: BlockingRecv error: recv got EOF from 127.0.0.1:41346 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 06:45:23.891386 30649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:23.889481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.7.146.253:35881)
0619 06:45:23.889679 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:23.889685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:23.889958 (+ 273us) server_negotiation.cc:407] Connection header received
0619 06:45:23.890191 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:23.890196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:23.890270 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:23.890360 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:23.891157 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41362: BlockingRecv error: recv got EOF from 127.0.0.1:41362 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 06:45:24.024991 30649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:24.022678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.7.146.253:35881)
0619 06:45:24.022896 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:24.022905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:24.023314 (+ 409us) server_negotiation.cc:407] Connection header received
0619 06:45:24.023547 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:24.023552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:24.023633 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:24.023730 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:24.024705 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41364: BlockingRecv error: recv got EOF from 127.0.0.1:41364 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:45:24.232829 30736 catalog_manager.cc:1259] Loaded cluster ID: be30b3de4199441aa13e99b7dd11a0b8
I20260619 06:45:24.232929 30736 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86: loading cluster ID for follower catalog manager: success
I20260619 06:45:24.235445 30736 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86: acquiring CA information for follower catalog manager: success
I20260619 06:45:24.237388 30736 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f53b362622e74c929406229880751a86: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:45:24.286492 30649 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:45:24.284728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.7.146.253:35881)
0619 06:45:24.284960 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:24.284966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:24.285114 (+ 148us) server_negotiation.cc:407] Connection header received
0619 06:45:24.285387 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:24.285391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:24.285467 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:24.285552 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:45:24.286282 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41366: BlockingRecv error: recv got EOF from 127.0.0.1:41366 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:45:24.608616 30784 catalog_manager.cc:1259] Loaded cluster ID: be30b3de4199441aa13e99b7dd11a0b8
I20260619 06:45:24.608754 30784 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20: loading cluster ID for follower catalog manager: success
I20260619 06:45:24.611380 30784 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20: acquiring CA information for follower catalog manager: success
I20260619 06:45:24.613334 30784 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1c705b4aac1b41d595300bbf65aa2e20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:45:24.815902 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:24.808691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.7.146.253:35881)
0619 06:45:24.809311 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:24.809318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:24.809532 (+ 214us) server_negotiation.cc:407] Connection header received
0619 06:45:24.809708 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:24.809714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:24.809910 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:24.810063 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:45:24.810529 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:24.811806 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:24.813211 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:24.813859 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:24.813995 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:24.814505 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:45:24.815261 (+ 756us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:45:24.815410 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:24.815463 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 06:45:24.815551 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":287,"threads_started":1}
I20260619 06:45:24.828961 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:24.822205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.7.146.252:45149)
0619 06:45:24.823008 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:24.823014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:24.823034 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:45:24.823147 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:24.823153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:24.823346 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:24.823554 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:45:24.823944 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:24.825048 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:24.826336 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:24.826951 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:24.827082 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:24.827608 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:45:24.828284 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:45:24.828458 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:24.828510 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 06:45:24.828645 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":589,"threads_started":1}
I20260619 06:45:24.856500 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:24.835005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.7.146.254:38291)
0619 06:45:24.847442 (+ 12437us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:24.847450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:24.848235 (+ 785us) server_negotiation.cc:407] Connection header received
0619 06:45:24.849166 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:24.849172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:24.849383 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:24.849535 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:45:24.850316 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:24.851344 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:24.853383 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:24.853998 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:24.854092 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:24.854975 (+ 883us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:45:24.855554 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:45:24.855664 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:24.855941 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 06:45:24.856070 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12266,"thread_start_us":12458,"threads_started":1}
I20260619 06:45:24.856724 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:24.835091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.7.146.253:35881)
0619 06:45:24.835270 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:24.835276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:24.848777 (+ 13501us) server_negotiation.cc:407] Connection header received
0619 06:45:24.848975 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:24.848981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:24.849075 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:24.849171 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:45:24.849912 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:24.851071 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:24.853324 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:24.854017 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:24.854099 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:24.854909 (+ 810us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:45:24.855054 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:45:24.855114 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:24.855379 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 06:45:24.855468 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:45:24.862113 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:45:24.848507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.7.146.252:45149)
0619 06:45:24.848733 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 06:45:24.848739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:45:24.855577 (+ 6838us) server_negotiation.cc:407] Connection header received
0619 06:45:24.856059 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:45:24.856065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:45:24.856162 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:45:24.856228 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:45:24.857074 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:24.858166 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:24.860349 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:45:24.860872 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:45:24.860963 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:45:24.861476 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:45:24.861589 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:45:24.861663 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:45:24.861808 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 06:45:24.861872 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 06:45:24.866670 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 30504
I20260619 06:45:24.867684 30566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:45:24.981597 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 30504
W20260619 06:45:24.998646 30595 connection.cc:535] server connection from 127.0.0.1:41234 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:45:24.998713 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 30578
I20260619 06:45:24.999814 30640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:45:25.108399 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 30578
I20260619 06:45:25.124718 7755 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu pid 30652
I20260619 06:45:25.125710 30714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:45:25.232867 7755 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjzr1uN/build/asan/bin/kudu with pid 30652
2026-06-19T06:45:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4741 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9594 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (350775 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST